Autonomous crop harvester

ABSTRACT

Systems and methods capable of selecting and positively grasping objects of interest within a cluttered environment are described. Some aspects of the present disclosure provide for real-time control of a robot that uses various sensors and reacts to sensor input in real time to adjust the robots path. In some embodiments, a robotic item picker includes an end effector having a shaft extending along a longitudinal axis between a proximal end and a distal end, a carriage configured to rotate about and translate along an intermediate portion of the shaft between a proximal position and a distal position, a suction device coupled to the distal end of the shaft, and a plurality of fingers spaced radially about the carriage. The robot may be a t-bot including a longitudinal member rotatable about a lengthwise axis of the longitudinal member, a carriage translatable along the lengthwise axis, and a radial member slidably mounted to the carriage. The end effector may be rotatably coupled at a distal end of the radial member.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 62/742,698, filed Oct. 8, 2018, titled AUTONOMOUS CROP HARVESTER,and U.S. Provisional Application Ser. No. 62/854,264, filed May 29,2019, titled AUTONOMOUS CROP HARVESTER, both of which are herebyincorporated by reference in their entirety and for all purposes.

FIELD

The present disclosure relates to robots, and more particularly toautonomous selection and retrieval of items using robots.

BACKGROUND

Industrial robots are frequently utilized in “pick and place”implementations for repeatedly retrieving and moving items. Existingsystems typically require items to be pre-sorted in predeterminedlocations such as bins or conveyors, and are not able to accurately andprecisely identify and retrieve items in cluttered environments. Manyexisting industrial robots are also not designed to pick and placedelicate, soft-bodied objects that can be easily damaged during the actsof picking up the object, moving the object to a new location, and/orreleasing the object into the new location. Robots that are designed tomanipulate easily-damaged objects typically include very precisemechanical specifications and small tolerances for error to achieve highaccuracy and near 100% success rate in picking and placing the objectwithout damaging it. Such robots tend to be heavy, costly machines thatare ill-suited for pick and place operations in industrial andagricultural environments and are unable to move through a field ofobjects with agility and precision to avoid damaging objects of interestduring picking operations. In addition, existing industrial robots arenot designed to pick up a plurality of delicate, differently shaped andsized, soft-bodied objects that are stationary and tethered to theground as the objects enter into a work space of the robot moving overthe plurality of objects. Accordingly, there is a need for improvementin many aspects of autonomous selection and retrieval of items usingrobots.

SUMMARY

The systems and methods of this disclosure each have several innovativeaspects, no single one of which is solely responsible for its desirableattributes. Without limiting the scope as expressed by the claims thatfollow, its more prominent features will now be discussed briefly.

In one embodiment, an end effector for a picking robot is described. Theend effector includes a shaft extending along a longitudinal axisbetween a proximal end and a distal end, a carriage configured to rotateabout and translate along an intermediate portion of the shaft between aproximal position and a distal position, a suction device coupled to thedistal end of the shaft, and a plurality of fingers spaced radiallyabout the carriage, each finger coupled to the carriage by a hinge andextending distally from the carriage, the plurality of fingersconfigured to envelop the intermediate portion of the shaft when thecarriage is in the proximal position, the plurality of fingers furtherconfigured to envelop a target object space including the distal end ofthe shaft and the suction device when the carriage is in the distalposition.

In some embodiments, the suction device is configured to apply a vacuumto the target object space when the carriage is in the proximal positionand in the distal position. In some embodiments, the suction device is adistal-most portion of the end effector when the carriage is in theproximal position, and the plurality of fingers are the distal-mostportion of the end effector when the carriage is in the distal position.In some embodiments, the plurality of figures do not envelop the distalend of the shaft when the carriage is in the proximal position. In someembodiments, the plurality of fingers are configured to envelop anobject within the target object space when the carriage is in the distalposition. In some embodiments, each of the plurality of fingers isconfigured to rotate about the hinge toward the longitudinal axis of theshaft. In some embodiments, a volume of the target object spacedecreases when each of the plurality of fingers rotates toward thelongitudinal axis of the shaft. In some embodiments, the carriage isrotationally fixed about the shaft, and wherein the shaft is rotatablymounted within the end effector such that the carriage is rotatable byrotating the shaft about the longitudinal axis. In some embodiments, theshaft includes a collar extending radially outward about the distal endof the shaft, each finger includes a finger extension extending radiallyinward toward the longitudinal axis at a proximal end of the finger,and, as the carriage moves from the proximal position towards the distalposition, the finger extension is configured to contact the collarbefore the carriage reaches the distal position. In some embodiments,movement of the carriage to the distal position following contact ofeach finger extension with the collar causes each finger to rotate aboutthe hinge toward the longitudinal axis. In some embodiments, movement ofthe carriage to the distal position following contact of each fingerextension with the collar causes the volume of the target object spaceto decrease. In some embodiments, each finger includes at least oneforce sensor configured to detect a force exerted between the finger andan object within the target object space. In some embodiments, the forceexerted between the finger and the object comprises at least one of anormal force and a shear force. In some embodiments, the suction deviceincludes at least one sensor configured to detect the presence of anobject within the target object space. In some embodiments, the at leastone sensor is selected from the group consisting of a linear positionsensor and a vacuum sensor. In some embodiments, the at least one sensorincludes a linear position sensor or a 3D force sensor, wherein theobject within the target object space is tethered to a stationaryobject, and wherein the at least one sensor is configured to detect adirection at which the object is tethered to the stationary object. Insome embodiments, the end effector further includes a camera coupled tothe end effector and processing circuitry configured to detect targetobjects based at least in part on images obtained from the camera. Insome embodiments, the processing circuitry is further configured todetermine target object locations with respect to a coordinate systembased at least in part on the images obtained from the camera. In someembodiments, the processing circuitry is further configured to share atleast one of image data and target object location data with processingcircuitry associated with a second end effector. In some embodiments,the suction device is a suction cup including a collapsible bellows.

In another embodiment, a method of picking a berry is described. Themethod includes, by a robotic process, engaging the berry with a suctiondevice of an end effector, moving a plurality of fingers of the endeffector from a retracted configuration to an extended configuration togrip the berry, moving the berry along a vertical linear path, movingthe berry along a circular arc defined by a stem of the berry as aradius, rotating the berry about a longitudinal axis of the berry, andmoving the berry upward along the vertical linear path to break thestem.

In some embodiments, the berry is moved upward to break the stem with anacceleration of at least 3 g. In some embodiments, the method furtherincludes calculating an azimuthal stem direction corresponding to thestem of the berry, wherein the circular arc is determined based on thecalculated azimuthal stem direction. In some embodiments, each of theplurality of fingers comprises at least one force sensor configured tomeasure a force exerted on the berry by the finger, and calculating theazimuthal stem direction includes detecting a change in force at theforce sensor in at least one of the fingers, and performing one or moretrigonometric operations based on the detected change in force todetermine an azimuthal stem direction. In some embodiments, each of theplurality of fingers includes at least one force sensor configured tomeasure a force exerted on the berry by the finger, and calculating theazimuthal stem direction includes, while moving the berry along thevertical linear path, detecting an increase in force at the forcesensors in two of the plurality of fingers, and determining an azimuthalstem direction based at least in part on known azimuthal directionscorresponding to the two of the plurality of fingers. In someembodiments, the berry is engaged based at least in part on apredetermined berry location. In some embodiments, engaging the berryincludes moving the end effector such that the suction device travelsalong at least a portion of a calculated ray approach path, the rayapproach path extending through and beyond the predetermined berrylocation; while moving the end effector, detecting engagement of theberry based on data received from at least one of a vacuum sensor and alinear position sensor associated with the suction device; and pausingmovement of the end effector based on detecting the engagement of theberry. In some embodiments, engaging the berry includes moving the endeffector such that the suction device travels along at least a portionof a calculated ray approach path to the predetermined berry location;while moving the end effector, determining an updated berry locationdifferent from the predetermined berry location based at least in parton image data from one or more imaging devices; calculating an updatedray approach path to the updated berry location; and moving the endeffector along at least a portion of the updated ray approach path toengage the berry. In some embodiments, the method further includesdetermining the predetermined berry location based at least in part onimage data from one or more imaging devices coupled to the end effector.In some embodiments, the method further includes determining thepredetermined berry location based at least in part on image data fromone or more imaging devices coupled to a second end effector differentthan the end effector that engages the berry. In some embodiments, themethod further includes determining the predetermined berry location byat least detecting a berry candidate including one or more red-coloredpixels in an image of a picking area, calculating a confidence levelbased on at least one of a count of the one or more red-colored pixels,a shape of a region defined by the one or more red-colored pixels, or ahue corresponding to at least one of the one or more red-colored pixels,and establishing, with reference to a coordinate system, thepredetermined berry location corresponding to the berry candidate, basedat least in part on the image.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned aspects, as well as other features, aspects, andadvantages of the present technology will now be described in connectionwith various implementations, with reference to the accompanyingdrawings. The illustrated implementations are merely examples and arenot intended to be limiting. Throughout the drawings, similar symbolstypically identify similar components, unless context dictatesotherwise.

FIG. 1 depicts an example embodiment of a harvester as described herein.

FIGS. 2A and 2B depict an example embodiment of a harvester work cellincluding a portion of a robot and an end effector of the harvester ofFIG. 1.

FIG. 3A is a cross-sectional view schematically illustrating componentsof a first example harvester, including a first example robot that canbe implemented on a harvester as described herein such as the harvesterof FIG. 1.

FIG. 3B is a cross-sectional view schematically illustrating componentsof a second example harvester, including a second example robot that canbe implemented on a harvester as described herein such as the harvesterof FIG. 1.

FIG. 3C is a cross-sectional view schematically illustrating componentsof a third example harvester, including the second example robot of FIG.3B in a horizontal configuration for picking vertically oriented items.

FIG. 4A is a high-level schematic block diagram illustrating hardwarecomponents of an example embodiment of a harvester as described herein.

FIG. 4B is a schematic block diagram illustrating hardware components ofthe harvester of FIG. 4A.

FIGS. 5A-5E depict an example end effector compatible with a robot asdescribed herein.

FIGS. 6A-6G depict an example sequence for picking a berry using theexample end effector of FIGS. 5A-5E.

FIG. 7 is a flow chart illustrating an example harvester-level methodfor picking a berry as described herein.

FIG. 8 is a flow chart illustrating an example end effector-level methodfor picking a berry as described herein.

FIG. 9A is a schematic overhead view of an example harvester including aLIDAR guidance system.

FIG. 9B is a cross-sectional view schematically illustrating an exampleconfiguration of a LIDAR relative to a furrow.

FIG. 10 is a flow diagram illustrating an example LIDAR control process.

FIGS. 11A-11D depict additional views of the end effector of FIGS. 5A-5Dduring the example sequence of FIGS. 6A-6F.

FIGS. 12A-12G depict an example robot including a modified t-botarchitecture for supporting and manipulating the example end effectorsdescribed herein.

FIGS. 13A-13D depict an example embodiment of a harvester as describedherein.

DETAILED DESCRIPTION

Embodiments of the present disclosure provide systems and methodscapable of selecting and positively grasping objects of interest withina cluttered environment. Implementations of the present disclosure canalso provide real-time control of a robot that reacts to sensor input inreal time to adjust the robot's path. Throughout the followingdescription, various embodiments will be described with reference to theexample implementation of picking or harvesting agricultural crops suchas strawberries. However, it will be understood that any of the systems,devices, or methods described herein may equally be applied to any otherrobotic, industrial, agricultural, or other application, for example,harvesting of other crops, handling of eggs or other delicate items,pick and place implementations, or the like.

Existing grippers are typically capable of picking items from acontrolled environment, such as a conveyor, bin, flat surface, tray, orthe like. However, existing gripping robots typically require the itemsto be well separated, sorted by type, or otherwise prepared in advancefor picking. Thus, existing gripping robots generally struggle toidentify, select, and/or pick items from a jumbled, crowded, orirregular environment, such as an environment in which items are closetogether, do not all have the same size and shape, are mixed with otherdebris or materials that are not intended to be picked, etc. Suchenvironments are challenging for successful pick and place operationsbecause locating and retrieving items of interest requires real-timeanalysis of many unpredictable, constantly-changing variables, such asbut not limited to visibility of items of interest that are present inmany different orientations and heights, and distinguishingcharacteristics of an item of interest from the surrounding environment(so that resources are not expended picking objects that are not ofitems of interest). In the case of grippers intended for use inagricultural or industrial applications, environmental conditions, suchas but not limited to changing terrain, moisture levels, time of day,temperature, lighting conditions, and density and inertia of debrissurrounding items of interest, are also unpredictable variables.

Some existing grippers that use suction cups may be able to engagespecific items due to their small size and the high accuracy of robotsused to manipulate these grippers. However, suction cup grippers do notpositively grasp items, and are therefore limited in the forces they canapply to those objects. These force limitations necessarily limit theaccelerations and therefore speed at which they can move objects. Theforce limitations also limit the mass of the objects they can move.

Embodiments of the present disclosure provide systems and methodscapable of selecting and positively grasping objects of interest withina cluttered environment. Gripping devices described herein use acombination of suction grasping and mechanical gripper grasping. Inthese embodiments, initial engagement of an item is accomplished with asuction cup that protrudes beyond gripper fingers when the fingers arein a retracted configuration. The retracted configuration gives theentire gripper assembly a high aspect ratio (for example, the gripperassembly is substantially longer than it is wide). Thus, the retractedconfiguration may advantageously allow the suction cup to reach downinto crowded environments, such as a jumbled, unpredictable mess ofleaves and stems, to engage a particular item, such as a berry to bepicked from within the mess of leaves and stems.

Some agricultural implementations may require a relatively highacceleration (e.g., in the range of 3 g-5 g or more), such as toseparate a berry or other fruit from a stem. Advantageously, embodimentsof the present disclosure do not rely solely on a suction cup, whichalone may not be able to apply sufficient force for a complete pickingoperation. Embodiments of the present disclosure advantageously includea suction cup that is capable of lifting an item of interest enough toat least partially move it clear of other surrounding items and debris,plus other features that advantageously disengage the item of interestfrom its surroundings and move the now-disengaged item of interest to anew location—all without damaging the item. Once the item is at leastpartially clear of surrounding items and debris, mechanical graspingelements such as fingers are actuated. The fingers can travel from theretracted configuration to an extended configuration. In the extendedconfiguration, the fingers can close around the suction cup and the itemengaged thereon, providing a positive grasp on the item. Thus, thegripping assemblies described herein may advantageously use initialsuction actions followed by gripping actions to allow the use of apositive mechanical grasp in cluttered environments that would otherwiseprevent reliable positive grasping of items.

In some aspects, the gripping assemblies described herein may furtherinclude one or more sensors configured to provide real-time feedback ofgrasping and/or picking operations. These sensors can allow the systemto terminate and/or modify a picking sequence based on new informationreceived from the various sensors in real-time. Real-time feedback maysave time when a full predetermined cycle to pick one particular item isaborted and not completed for each item picking sequence in a series ofitem picking sequences. Real-time adjustment may also be advantageous bycontributing to successful operation in a less structured or moreirregular environment. In one non-limiting example, the robot workvolume is a field of strawberry plants with thousands of individualstrawberries, many of which are ready for harvest, which translates tothousands of individual item picking sequences. Relatively small timesavings (for example, one second, a half second, etc.) on a percentageof these sequences due to real-time feedback thus results in significanttime and cost savings. For example, although operating costs may vary byembodiment, in some cases an average time reduction of one second perpick sequence may result in a profitable operation, relative to a picksequence that may not even break even relative to manual harvesting.

In one example of the present disclosure, the robot or an end effector(for example, a gripping assembly) coupled to the robot includes avacuum sensor that can detect when the suction cup has generated a sealagainst an item of interest. The vacuum sensor may report an analogvalue that may be used to determine the quality or strength of the sealbased on how much vacuum (e.g., negative pressure) is reported. Inoperating environments that are cluttered and/or have non-static itemsto be picked, the estimate of an item's position is not always correctdue to the presence of occluding debris, motion of the items, and/orimperfect estimates of the initial location of an item. Thus, thesuction cup may make contact with an item before or after it is expectedto make contact due to movement or an inaccurate assessment of theitem's position. In view of this, the robot's suction mechanism may beconfigured to follow a path extending through and beyond the item'sestimated position. However, once a vacuum is detected at the vacuumsensor, a processor controlling the robot may stop forward motionbecause the detected vacuum indicates that the suction cup has madecontact with the item, or is very likely to have made contact with theitem with a high degree of confidence.

In another example, the suction cup (or a vacuum generator or otherstructure to which the suction cup is attached) may include a pushswitch or other positional detector. A positional detector may allow thesystem to detect when the gripping assembly has made contact with or“run into” something, such as an item or other debris. Thus, rather thancontinuing through a fully predetermined sequence, the gripping assemblymay detect if it is running into an item and can stop forward motion inresponse to this detected contact. Thus, a positional detector, alone orin combination with a vacuum sensor, can advantageously allow a grippingassembly to reach for items without the system controlling the grippingassembly having exact certainty of where the items are, by planning totravel through an item based on assessment of the item's likely positionbut stopping once contact is made (or very likely to have been madewithin a certain degree of confidence).

In a third example, some or all of the fingers in a gripping assemblycan each include one or more force sensors within the finger. A forcesensor can be positioned to detect the force exerted by a single fingeron an item being grasped by a plurality of fingers. Thus, force sensorsmay be used to detect when an item is being grasped and, based onreal-time feedback from the sensor, avoid exerting too much graspingforce against a delicate item, such as a berry. Moreover, if the item ofinterest is tethered to a stationary object (for example, a berry on astem connected to a bush planted in the ground), the forces detected atthe force sensors within the three fingers can be used to determine thedirection from which the tether is pulling against the grippingassembly's motion. More specifically, when the gripper is moved awayfrom the initial grasp location, the corresponding changes in the forcesensor measurements can be used to determine the stem or tetherdirection based on known trigonometric principles. As will be describedin greater detail below, picking efficiency, success rate, andsensitivity of the grasping and disengagement operations to minimizedamage may be improved when the stem or tether direction is known. In afurther example, the force sensors in the fingers may be advantageouslyused to detect when a grasped item has been dropped and, when anaccidental drop is detected, stop and re-attempt to pick up the itemrather than waste time by continuing to a pre-determined drop locationwithout the item.

The harvesters described herein may use eye-in-hand configurations inwhich some or all of the imaging cameras used to detect and identifyitems for picking are mounted on the moving end effectors, rather thanin a static location within the system. Eye-in-hand configurations ofthe present disclosure have several advantages for pickingimplementations. For example, cameras mounted on implementations of endeffectors of the present disclosure can look around objects such asfoliage or the end effector itself, that would otherwise partially blockimages taken from a static viewpoint. In another example, during apicking sequence of a first object of interest, end effector-mountedcameras may be able to identify several possible ray approach paths thatcan later be used by an end effector to approach a second item ofinterest for picking without running into debris or clutter in thevicinity of the item, thereby increasing efficiency, success rate,and/or sensitivity of the second picking sequence. In yet anotheradvantage, eye-in-hand configurations may allow for less precise orwell-calibrated cameras to be used. For example, an error in itemposition estimation due to camera calibration may be relatively largefor a camera located above the end effector, but may be significantlysmaller for a camera located on the end effector near the item. Inaddition, analysis of image data may be simplified because positioning acamera on the end effector may eliminate the need to accuratelycalculate the location of the end effector based on image data.

Example Harvester According to the Present Disclosure

FIG. 1 depicts an example embodiment of a harvester 100 as describedherein. The harvester 100 includes wheels 105 supporting a platform 110.The harvester 100 further includes one or more work cells 115, each workcell 115 including one or more robots 120. Each robot 120 is configuredto support and move an end effector 125 for picking and removing itemswithin a work volume. The work volume is a region of three-dimensionalspace, generally below the harvester 100, in which the robot or robotsof a work cell 115 are configured to move and pick items. In someembodiments, the work volume can include a volume of space at and abovethe surface of an agricultural field, including but not limited to afield of strawberry plants. The wheels 105 can be sized, shaped, andspaced so as to sit within furrows 50 of an agricultural field such thatthe platform 110 supports the work cells 115, robots 120, and endeffectors 125 over one or more rows 55 disposed between the furrows 50.In the example implementation depicted in FIG. 1, the rows containstrawberry plants including at least some ripe strawberries to bepicked. It will be understood that embodiments of the harvester 100 arenot limited to moving in the work volume using wheels. Any othersuitable mechanism can be employed, such as but not limited to a slidingsled mounted on rails running in the furrows or an elevated platformmoved on a cable suspended over the work volume.

The harvester 100 may further include one or more motors under controlof processing circuitry and configured to drive one or more of thewheels 105 to propel the harvester 100 along the furrows 50. The motorsand processing circuitry may be configured to propel the harvester 100at a constant or substantially constant speed, and/or may be configuredto propel the harvester 100 at a variable speed. For example, in someembodiments the harvester 100 may travel more slowly or stop when arelatively large number of items have been identified to be pickedwithin the working volume of one or more work cells 115, and may travelat a relatively higher speed when fewer items have been identified to bepicked within the working volumes of the work cells 115.

FIGS. 2A and 2B depict close-up views of a portion of the harvester 100of FIG. 1. The views of FIGS. 2A and 2B illustrate a work cell 200,including a receptacle for picked items 215 and a portion of a robot 120having arms 205 and an end effector 210. As described with reference toFIG. 1, the harvester 100 supports the robot 120 and the end effector210 over a row 55 including plants 60. The robot 120 and the endeffector 210 are movable within the work volume relative to theharvester 100. The views in FIGS. 2A and 2B are images of individualpoints in time during picking operations. However, it will be understoodthat the work volume moves in space relative to the harvester 100 as theharvester moves along the rows 55, adding to the complexity of thepicking methods described herein.

In the example harvester 100 of FIGS. 2A and 2B, the robot 120 is adelta robot configured to move the end effector 210 using arms 205. Insome embodiments, delta robots may be advantageous for the systemsdescribed herein as they can provide relatively high speed and/oracceleration, as well as high stability for supporting the end effector210. In other implementations, a t-bot or any other suitable type ofrobot, robotic arm, or the like may be used to support and position theend effector 210 within the harvester 100.

While the work cell 200 is disposed above the row 55, individual itemssuch as berries may be picked from the plants 60 by the robot using theend effector 210. As shown in FIG. 2A, the arms 205 may position the endeffector 210 above a plant 60 containing an item of interest to bepicked. Once the end effector 210 has grasped and picked the item, thearms 205 move the end effector 210 and the item to a location above thereceptacle 215, as shown in FIG. 2B. When the end effector 210 is abovethe receptacle 215, the end effector 210 releases the item such that theitem falls into the receptacle 215. In some embodiments, overallharvesting speed may be increased by causing the end effector 210 torelease the item before reaching a location above the receptacle 215,while traveling toward the receptacle 215, such that the item's velocityand trajectory carries it across the remaining lateral distance to fallinto the receptacle 215.

FIG. 3A is a cross-sectional view schematically illustrating componentsof an example harvester 300 according to the present disclosure. Similarto the harvester 100 of FIGS. 1, 2A, 2B, the harvester 300 includes aplurality of wheels 305 supporting a platform 310. The harvester 300includes at least one work cell 315, the at least one work cellincluding at least one robot 320. The example harvester 300 illustratedin FIG. 3A includes one work cell 315 that includes two robots 320, eachrobot 320 having a plurality of arms 322 configured to support andmanipulate an end effector 325 for picking. However, more than tworobots 320 may also be included in a work cell 315. As will be describedin greater detail below, multiple robots 320 operating within a singlework cell 315 may be in communication such that they can avoid mutualinterference while moving. In addition, multiple robots 320 within awork cell 315 may exchange information regarding location andacquisition of target items to enhance the accuracy and/or efficiency ofidentifying and picking items.

The harvester 300 is configured to travel within an agricultural field.Accordingly, the platform 310 is positioned relative to the wheels 305such that the platform 310 is supported above a row 55 while the wheels305 travel along the bottom of furrows 50 surrounding the row 55.Although the harvester 300 is depicted as having a single work cell 315supported by wheels 305 spaced to travel in adjacent furrows 50, largerharvesters 300 including more than one work cell 315 can equally beimplemented using the systems and methods described herein. For example,a harvester 300 may have a wider wheel base such that the harvesterspans two, three, four, or more rows 55. Generally, such harvesters 300may have an individual work cell 315 for each row, and may furtherinclude additional wheels 305 spaced to travel in intermediate furrows50 to support the frame 310. In one particular example, a harvester 300is sized to span three rows 50, and thus includes three work cells 315spaced along the x-axis, each work cell 315 including two robots 320 andtwo end effectors 325. In another example, a harvester 300 may includemultiple work cells 315 spaced along the y-axis and configured tooperate on a single row, for example, a forward work cell 315 and an aftwork cell 315. In this example, the forward work cell 315 may pick onlya portion (for example, half) of the items of interest in the workvolume, while the aft work cell picks the remaining items of interestfrom the work volume after the forward work cell 315 has conducted afirst pass.

The harvester 300 may be propelled autonomously along the furrows 50. Insome embodiments, at least one guidance sensor 330 is coupled to theharvester 300 to detect the contour of the furrow 50. In variousimplementations, the guidance sensor 330 may include any one or more ofa LIDAR system, an optical detector such as a camera, or the like. Aguidance system of the harvester 300 may steer one or more wheels of theharvester 300 such that the wheels 305 remain within and do not impactthe sides of the furrows 50. Light sources 335 may further be providedto illuminate at least a portion of the row 55 and/or furrows 50 tofacilitate picking and/or guidance operations. Harvester guidance isdescribed in greater detail below with reference to FIGS. 9A-10.

FIG. 3B schematically illustrates a further example of a harvester 300.While the robots 320 of FIG. 3A were illustrated as delta robots, therobots 320 of FIG. 3B are illustrated as modified t-bots. In someembodiments, a t-bot or modified t-bot may advantageously be smallerthan a delta robot and/or may include fewer components capable ofextending toward components of an adjacent robot. Thus, the modifiedt-bot implementations described herein may advantageously reduce theprobability of a collision occurring between multiple robots of aharvester, may reduce and/or simplify the computational requirements forpreventing collisions between robots, may reduce interference with otherobjects within the picking area (e.g., branches and the like), mayprovide a high aspect ratio suitable for reaching into plants or othercluttered areas, and may include various other operational advantages aswill be described in greater detail with reference to FIGS. 12A-12G.

FIG. 3C schematically illustrates an additional example of a harvester300. As shown in FIG. 3C, the robots 320 of FIG. 3B may be oriented in ahorizontal configuration instead of or in addition to the verticalconfiguration illustrated in FIG. 3B. In some embodiments, a horizontalconfiguration may allow the harvester 300 to be used for pickingvertically grown crops such as vine crops, tree crops, or verticallygrown arrangements of crops such as strawberries or other conventionallyhorizontal crops. The horizontal configuration may further allow theharvester 300 to be used for pruning implementations.

FIG. 4A is a high-level schematic block diagram illustrating a hardwarehierarchy of an example harvester 400. The harvester 400 includes avehicle master controller 402, a communications module 404, a GPS module406, an item storage container 408, a LIDAR system 410, a vehiclecontrol unit 412, a user interface 414, and any number of work cells 416₁, 416 ₂, . . . 416 _(N). Each work cell 416 ₁, 416 ₂, . . . 416 _(N)includes an air compressor 418, lighting 420, and two robots 422 ₁, 422₂, although in some embodiments work cells may contain a single robot ormore than two robots. Although only illustrated for work cell 416 _(N),it will be understood that the other work cells may each include similarcomponents. Each robot 422 ₁, 422 ₂ includes an end effector and servos424 ₁, 424 ₂, 424 ₃ for moving the end effector. Although varioussubcomponents are only depicted for robot 422 ₁, any other robots withinthe harvester 400 may include similar components. The end effectorincludes at least a slave controller in communication with an endeffector hub 428 and an end effector microcontroller 440. The endeffector hub 428 is in communication with any input devices of the endeffector, including one or more cameras 430 ₁, 430 ₂, 430 ₃, and asensor hub 432. Some or all of the cameras 430 ₁, 430 ₂, 430 ₃ may bedisposed on the end effector (e.g., eye-in-hand configurations) orelsewhere, such as on a robot 422 ₁, 422 ₂, or on another moving orstationary component of the harvester 400. The sensor hub 432 receivesdata from a vacuum sensor 434, a touch sensor 436, and finger sensors438 located within the end effector. The microcontroller 440 isconfigured to actuate a suction control 442, finger extension control444, and rotation control 446 of the end effector. Although variousprocessing components are described as being distributed among variouscontrollers, microcontrollers, hubs, and the like, it will be understoodthat any of these components and/or processing operations may bedifferently grouped and/or may be implemented in more or fewercomponents than those depicted in FIG. 4A.

The vehicle master controller 402 is in communication with each workcell 416 ₁, 416 ₂, . . . 416 _(N) in the harvester. The vehicle mastercontroller is also in communication with harvester-level componentsincluding the communications module 404, the GPS module 406, the LIDARsystem 410, the vehicle control unit 412, and the user interface 414.The vehicle master controller 402 can control the movement and/ornavigation of the harvester 400, for example, based on informationreceived from the work cells 416 ₁, 416 ₂, . . . 416 _(N), the robots422 ₁, 422 ₂, or other components. As described with reference to FIG.3, the vehicle master controller 402 may use ground contour informationfrom the LIDAR system 410 to cause the vehicle control unit 412 to steerthe harvester 400 such that the wheels of the harvester remain withinfurrows and do not collide with planting rows. The vehicle mastercontroller 402 may use location data from the GPS module 406 to executea predetermined harvesting path, report the harvester's location via thecommunications module 404, or perform other position-related functionssuch as yield logging/recording, plant health logging/recording, blossomcount logging/recording, etc. The vehicle master controller 402 mayreport other metrics (e.g., key performance indicators). For example,the vehicle master controller 402 may report metrics such as berriespicked per robot, current pick rate (e.g., in seconds per berry), orother operational information associated with the harvester 400. Thecommunications module 404 may be configured to communicate by one ormore wireless communication paths, such as a cellular network, mobileinternet (e.g., 3G, LTE, etc.), Bluetooth, near field communication(NFC), Wi-Fi, or the like.

The harvester may have any number of work cells 416 ₁, 416 ₂, . . . 416_(N). Each work cell 416 ₁, 416 ₂, . . . 416 _(N) may include adedicated compressor 418 for supplying pressurized air for pneumaticsystems, and lighting 420 to illuminate the picking area of the workcell 416 ₁, 416 ₂, . . . 416 _(N). If the work cells 416 ₁, 416 ₂, . . .416 _(N) each include more than one robot 422 ₁, 422 ₂, the robots 422 ₁and 422 ₂ of the work cell may be in communication with each other toavoid colliding during picking operations. Each robot 422 ₁, 422 ₂includes one or more servos 424 ₁, 424 ₂, 424 ₃ for moving the endeffector within the work cell area. In the example of a delta robot,three servos may be used. In the example t-bot embodiments describedherein, three or more servos may be used, for example, depending on thenumber of axes included in the particular t-bot design.

Consistent with the hierarchy of FIG. 4A, FIG. 4B is a schematic blockdiagram illustrating electrical connections and hardware components ofan example implementation of a harvester 400 according to oneimplementation of the present disclosure. FIG. 4B schematicallyillustrates a harvester 400 including three work cells 416 ₁, 416 ₂, 416₃. Although only the components of work cell 416 ₃ are shown in FIG. 4B,it will be understood that work cells 416 ₁ and 416 ₂ can include all ora subset of the components illustrated within work cell 416 ₃. A networkswitch 401 provides for switchable communications between each of thework cells 416 ₁, 416 ₂, . . . 416 _(N) and harvester-level componentssuch as the vehicle master controller 402, the LIDAR system 410, and thecommunications module 404. The vehicle master controller 402 is directlyor indirectly in electrical communication with the vehicle control unit412, motor controller 411, vehicle control encoders 413, vehicle controlinput 415, and user interface control 414. The vehicle control unit 412may be a microcontroller or other processing circuitry configured toreceive information from the vehicle control encoders 413 and vehiclecontrol input 415. The vehicle control unit 412 may further beconfigured to command the motor controller 411. The motor controller 411may be electrical drivers or the like, configured to rotate one or moredrive motors using system power to move the harvester 400. The vehiclecontrol encoders 413 may be, e.g., rotary encoders or the like,configured to communicate angular position and/or speed of one or moredrive motors of the harvester 400. The vehicle control input 415 is adevice for receiving manual control commands, such as a joystick or thelike. In various embodiments, the vehicle control unit 415 may be usedfor diagnostic operation, initial position, storage, or othernon-autonomous aspects of harvester 400 operation. The user interfacecontrol 414 may include additional manual controls such as one or morebuttons, an on/off switch and/or key, status indicator or annunciatorlamps, or the like. Certain embodiments of the harvester 400 may notinclude a vehicle control unit 415 and/or user interface control 414,and may be configured to operate autonomously without input from thevehicle control unit 415 and/or user interface control 414.

A work cell-level network switch 417 is included within each of the workcells 416 ₁, 416 ₂, . . . 416 _(N) to route communications between theharvester-level network switch 401 and components of the work cell 416₁, 416 ₂, . . . 416 _(N). In this example implementation, each work cellincludes two robots 422 ₁, 422 ₂ in communication with the networkswitch 417. Within each robot 422 ₁, 422 ₂, a slave controller 426provides connectivity between the work cell-level network switch 417 andthe end effector hub 428 and robot controller 440. The end effector hub428 is in communication with and configured to receive sensor data fromone or more vacuum sensors 434, linear position sensors 436, and fingerforce sensors 438.

The end effector hub 428 is further in communication with cameras 430 ₁,430 ₂, 430 ₃, some or all of which may be mounted to the end effector inan eye-in-hand configuration. Lighting elements 431 ₁, 431 ₂, 431 ₃,which may be located adjacent to corresponding cameras 430 ₁, 430 ₂, 430₃, are in communication with the robot controller 440, which isconfigured to control the activation and/or intensity of the lightingelements 431 ₁, 431 ₂, 431 ₃. In some embodiments, the robot controller440 may be configured to control the lighting elements 431 ₁, 431 ₂, 431₃ based at least in part on image data received from the cameras 430 ₁,430 ₂, 430 ₃ at the end effector hub 428. For example, the intensity oflight produced at an individual lighting element (e.g., element 431 ₁)may be reduced when the overall light intensity detected at thecorresponding camera 430 ₁ is higher than a predetermined threshold, orwhen it is determined that an object is in close proximity to the camera430 ₁. In some embodiments, the robot controller 440 may be configuredto control the lighting elements 431 ₁, 431 ₂, 431 ₃ based at least inpart on one or more light sensors located near the cameras 430 ₁, 431 ₂,430 ₃, and/or based on detected proximity to an object.

The robot controller 440 is further in communication with, andconfigured to at least partially control, various other switchable orcontrollable robot components, such as one or more relays 419 forcontrolling an air compressor 418 and/or vision lights 420, one or moresolenoids or other switches for controlling actuators 423 within the endeffector, and the servos 424 for controlling the robot to move the endeffector.

Electrical power for operation of the harvester 400, work cells 416 ₁,416 ₂, 416 ₃, and robots 422 ₁, 422 ₂ can be received from a systembattery 466 and/or a generator 464. Additional components such as astarter 472 for the generator 464 and a battery charger 468 for chargingthe system battery from the generator 464 may be provided. An inverter462 and one or more harvester-level circuit breakers 460 can be incommunication with the generator and system battery to provideelectrical power to each work cell 416 ₁, 416 ₂, 416 ₃. An emergencystop switch 470 may be provided to allow an operator to cut off power tothe harvester 400.

Within each work cell 416 ₁, 416 ₂, 416 ₃, a work cell main circuitbreaker 450 is connected to the harvester-level circuit breaker 460 aswell as any auxiliary circuit breakers 452. An AC/DC converter 451within the work cell provides DC electrical power to the robotcontroller 440. A work cell-level battery charger 454 can charge a workcell battery 456. The work cell battery 456 may be connected to one ormore DC/DC converters 458 to provide DC electrical power at one or moredesired voltages as necessary to operate the various electricalcomponents within the work cell 416 ₁, 416 ₂, 416 ₃. Various embodimentsmay include alternative power supply configurations, for example,including features such as thermal circuit breakers, contactors or otherrelays, emergency stop circuitry, or the like.

Example End Effector According to the Present Disclosure

FIGS. 5A-5D depict an example end effector 500 or gripper according toone implementation of the present disclosure. The end effector 500 isconfigured to operate in conjunction with a robot to identify, select,and pick items. As will be described in greater detail, the end effector500 includes various features that may provide enhanced utility for thepicking of delicate items, such as berries or other agriculturalproducts, as well as for picking items in a cluttered and/or non-staticenvironment. The end effector 500 may operate at least partially underthe control of a controller 501 (which can be, for example, slavecontroller 426 described above with reference to FIG. 4A) and/or otherprocessing components. The end effector 500 in this example includes abase 510, a shaft 520, and a carriage 530 coupled to a plurality offingers 540. Although the end effector 500 is illustrated as havingthree fingers 540, some embodiments may include two fingers, fourfingers, five fingers, or more.

The base 510 serves as a mounting platform to which other components ofthe end effector 500 are coupled. One or more robot arms 512, a shaftactuator gear 514, and a shaft actuator 515 are coupled to the base 510.A partial section of six robot arms 512 is illustrated in FIG. 5A (whichcan be, for example robot arms 322 described above with reference toFIG. 3). FIGS. 5A-5E depict six robot arms 512 but it will be understoodthat any suitable number of robot arms can be implemented. The base 510may include one or more generally rigid materials, such as a metal, ahard plastic, or the like. The base 510 includes ball-socket typeconnections 511 to six robot arms 512 for control by a delta robot,which may include three pairs of two arms. A delta robot may bedesirable for operation of the end effector 500, as a delta robot may bewell-suited for keeping the end effector 500 in a constant vertical ornear vertical orientation. However, any other suitable type of roboticarm or support structure can be implemented in embodiments of thepresent disclosure, and it will be understood that the locations ofconnections on the base 510 may vary for other robot configurations, forexample, as shown in FIGS. 12A-12G.

The shaft 520 extends perpendicularly downward from the base 510 along alongitudinal axis 521 and provides a structure along which the carriage530 and fingers 540 may be translated longitudinally along the axis. Theshaft 520 includes a proximal end 527 _(P), a distal end 527 _(D), andan intermediate portion 527 _(I). The base 510 is generally located atthe proximal end 527 _(P) of the shaft. In some embodiments, the shaft520 may extend above the base 510 as well. A suction device such as asuction cup 522 is coupled to the distal end 527 _(D) of the shaft 520by a suction cup coupler 524. The suction device may be the suction cup522 having collapsible bellows, but should not be limited to thisembodiment and may include any of various other suction devices. Thedistal end 527 _(D) of the shaft 520 further includes a collar 526. Ashaft gear 528 for rotation of the shaft 520 is located at the proximalend 527 _(P) of the shaft. The shaft gear 528 is substantially coaxialwith the shaft 520 and may be formed integrally with the shaft 520 ormay be formed as a distinct component and joined to the shaft 520. Inembodiments with a separately formed shaft gear 528, it may be desirablefor the shaft 520 to have a non-circular cross section (e.g., thehexagonal cross-section depicted in FIGS. 5A-5D) to allow rotation ofthe shaft gear 528 to induce rotation of the shaft 520.

The suction cup 522 is disposed at the distal end of the shaft 520 suchthat, when the carriage 530 and fingers 540 are retracted toward theproximal end of the shaft 520, the suction cup 522 is the lowest pointof the end effector 500 in the z-axis direction and can be used toinitially engage an item to be picked. One or more pneumatic connections525 provide pressurized air and/or suction for operation of the suctioncup 522. In some embodiments, the suction cup 522 may include and/or maybe connected to a Venturi effect suction device, and the pneumaticconnections 525 may provide pressurized air to the suction cup 522. Thepneumatic connections 525 may be routed through the interior of theshaft 520 to the suction cup 522. In the non-limiting embodimentdepicted in FIGS. 5A-5D, the suction cup 522 includes a collapsiblebellows configured to conform to the surface of an item of interest whenengaging the item.

The suction cup 522 may further include a vacuum sensor (e.g., apressure sensor) configured to detect a pressure in the space within thesuction cup 522. In some embodiments, the Venturi effect device or othernegative pressure device may continuously create a negative pressure atan opening 523 of the suction cup 522. As long as the opening 523 isopen to the atmosphere and is not obscured by an item, the vacuum sensormay detect a consistent pressure similar to or slightly lower thanatmospheric pressure. However, when the opening 523 is obscured orblocked (as, for example, when the suction cup 522 engages with thesurface of an item), the pressure within the suction cup 522 may dropsubstantially lower due to the operation of the negative pressuredevice. Accordingly, a sudden low-pressure detection at the vacuumsensor may indicate to the controller 501 or other processing circuitrythat the suction cup 522 has contacted an item. Varying degrees ofpressure measured by the vacuum sensor can also be used to indicatedegree and sufficiency of engagement with the contacted item. In thenon-limiting example of strawberry picking, an optimal engagement with arelatively flat and smooth side of a berry may result in a lowestpressure detection, while engagement with the calyx or a combination ofthe berry and the calyx may result in a less effective seal and acorresponding higher pressure measurement. Other scenarios, such as whena relatively small berry is sucked up into the bellows or when a stem orleaf is positioned between the suction cup 522 and the body of theberry, may also be detected when a less optimal seal causes a reducedvacuum. In some embodiments, a negative pressure device located near thetip of the shaft 520 may advantageously enhance detection at the vacuumsensor. For example, where the vacuum cavity is small (e.g., includingonly the tip of the shaft 520, rather than the entire interior of theshaft 520), evacuation of the cavity occurs relatively quickly when theopening 523 is blocked, resulting in a shorter time required to detectblocking of the opening 523.

The suction cup 522 may be coupled to the shaft 520 at the suction cupcoupler 524. In some embodiments, the suction cup coupler 524 may bemovably coupled to the shaft 520. In one example, the suction cup 522 isfixedly coupled to the suction cup coupler 524, and the suction cupcoupler 524 is slidably mounted within the shaft 520 such that an upwardforce at the opening 523 of the suction cup 522 causes the suction cupcoupler 524 to move upward along the longitudinal axis 521 relative tothe shaft 520. In this example, a linear position sensor included withinthe end effector 500 may be triggered by the motion of the suction cupcoupler 524. Accordingly, the triggering of the linear position sensormay indicate to the controller 501 or other processing circuitry thatthe suction cup 522 has contacted an item. The suction cup coupler 524may comprise a portion of the vacuum generator associated with thesuction cup 522. For example, the suction cup coupler 524 may include aVenturi effect device or other vacuum generator sized and shaped tosecurably receive the suction cup 522.

The collar 526 includes a portion of the shaft 520 at the distal endthat is wider (e.g., has a greater radial extent) about the longitudinalaxis 521 relative to a portion of the shaft 520 above the collar 526.The collar 526 may be integrally formed with the shaft 520 or may beformed separately and joined to the shaft 520, and may comprise the samematerial as the shaft 520 or any other substantially rigid material. Byhaving a larger radial extent relative to the proximal portion of theshaft 520, the collar 526 can retain components that are slidablymounted on the shaft 520 and prevent such components from sliding offthe end of the shaft 520. Such components can include, for example, thecarriage 530. In the example end effector 500, the collar 526 has acircular cross-section about the longitudinal axis 521 so as to retainshaft-mounted components at any radial location about the longitudinalaxis 521.

The shaft gear 528 is rotationally fixed relative to the shaft 520, suchthat rotation of the shaft gear 528 causes the shaft 520 to rotate aboutthe longitudinal axis. Thus, the shaft gear 528 can be rotated to rotatethe shaft 520 and any other components rotationally fixed thereto, suchas the carriage 530 and fingers 540. In the example end effector 500,the shaft gear 528 can be coupled to the shaft actuator gear 514 by abelt, chain, or any other suitable mechanism, such that rotation of theshaft actuator gear 514 causes the shaft gear 528 and the shaft 520 torotate about the longitudinal axis. A non-coaxially located shaftactuator 515 (which may include the rotation control 446 of FIG. 4A) canthus rotate the shaft 520 by rotating the shaft actuator gear 514. Theshaft actuator 515 may be, for example, a pneumatic rotary actuator, aservomotor, or other suitable actuator. Rotation of the shaft 520relative to the end effector 500 may be controlled by processingcircuitry of the shaft actuator 515 and/or the controller 501, which mayinclude the slave controller 426 of FIG. 4A, the microcontroller 440,and/or other processing components.

The carriage 530 is disposed about the shaft 520 and supports thefingers 540. The carriage 530 may include any substantially rigidmaterial, such as but not limited to a metal or a hard plastic. Thecarriage 530 is rotatably coupled to a carriage slider 532 in alongitudinally fixed configuration along the longitudinal axis. Thecarriage slider 532 is fixed to one or more rods 534 driven by linearactuators 535 to control the longitudinal position of the carriage 530along the shaft 520 and/or a controllable force applied to the carriage530 (e.g., to control a grasping force applied at the fingers 540, aswill be described below in greater detail). Linear actuators 535 may beany suitable type of linear actuator, such as mechanical, pneumatic,hydraulic, electro-mechanical, etc. The carriage 530 further includeshinge points 536 for rotatably coupling the fingers 540 to the carriage530.

In order to provide for longitudinal and rotational collective motion ofthe fingers 540, the carriage 530 is rotatable about the longitudinalaxis and is translatable along the longitudinal axis. To achieve thesetwo degrees of motion, the carriage 530 can be rotationally fixed andlongitudinally-movable relative to the shaft 520, and can berotationally-movable and longitudinally fixed relative to the carriageslider 532. In some embodiments, the carriage 530 has a central openinghaving a complementary profile to the cross-section of the shaft. Forexample, for a shaft 520 having a hexagonal cross-section as shown inFIGS. 5A-5D, the carriage 530 may have a hexagonal central opening sizedand shaped to engage with the hexagonal cross-section of the shaft 520such that rotation of the shaft 520 about the longitudinal axis causesthe carriage 530 to rotate simultaneously. The carriage 530 may becoupled coaxially with the carriage slider 532 by a bearing 538 (forexample, a plain bearing, a ball bearing or other suitablerotating-element bearing, etc.) to allow the carriage 530 to rotatefreely relative to the carriage slider 532 while fixing the carriage 530longitudinally to the carriage slider 532.

The fingers 540 are coupled to the carriage 530 and provide a positivemechanical grasp on items engaged at the suction cup 522. Each finger540 can include a substantially rigid base 542 formed of a hard plastic,a metal, or any other suitable material. Each finger can also include aresilient and/or cushioned tip 544 formed of a soft plastic, a rubber,or any other suitable elastomeric material. For example, the tip 544 maycomprise a cushioned or resilient material on at least an inward-facinggrasping surface 545. The base 542 of each finger 540 can be mounted tothe carriage 530 at a hinge point 536, such that the fingers 540 canrotate inward toward the suction cup or outward away from the suctioncup about the hinge points 536. In some embodiments, the tip 544 formsmost of the exterior surface of the finger 540. For example, the rigidbase 542 may include a central spine on which the resilient tip 544 maybe formed or adhered. In some embodiments, the resilient tip 544 may bea slidably removable tip that can be attached and/or removed by slidingthe resilient tip 544 onto or off of the rigid base 542. The rigid basemay include one or more locking features (not shown) that may engagewith corresponding features of the resilient tip 544 to detachablysecure the resilient top 544 onto the rigid base 542. Thus, the fingersmay provide cushioned and/or compressible interior engagement surfaceswhile remaining securely fixed to the carriage 530.

The fingers 540 are radially spaced about the carriage to define atarget object space surrounding the suction cup 522 and bounded by theinterior surfaces 545 of the fingers 540 when the carriage 530 islocated at or near the distal end of the shaft 520. The fingers 540 mayrotate inward about the hinge points 536 to engage an item locatedwithin the target object space. This inward rotation decreases thevolume of the target object space, allowing the fingers to close in onthe target object space until they contact and grasp the item engaged bythe suction cup. This can allow for reliable grasping of the item, nomatter what orientation or position the item is in relative to thesuction cup. The fingers 540 may be individually or collectivelyactuated to rotate about the hinge points 536. In some embodiments, thefingers 540 may be indirectly actuated such that additional linear orrotational actuators need not be provided in the vicinity of thecarriage. In one non-limiting example embodiment, the fingers 540 may bemechanically interlocked with the shaft 520 such that they automaticallyclose around the target object space as the carriage 530 approaches itsextreme distal position.

Each finger 540 is outwardly biased (e.g., by a spring) to an openconfiguration in example implementations. Each finger 540 furtherincludes a substantially rigid extension 546 that extends radiallyinward from the base 542. In this example, as the carriage 530 andfingers 540 travel toward the distal end along the shaft 520, a distalside of each extension 546 will contact the proximal-most surface ofcollar 526 while the carriage 530 is still at an intermediate positionproximally spaced from its extreme distal position. As the carriage 530moves beyond this intermediate position, the extensions 546 will beretained in the same longitudinal position along the shaft 520 by thecollar 526 and any further distal motion of the carriage 530 will causethe fingers 540 to rotate about the hinge points 536. In this manner,control of the end effector 500 may be simplified as the linearactuators 535 may be able to control both translation of the carriage530 and closing of the fingers 540 within a single axis of motion.

Some or all of the fingers 540 may include one or more force sensors atan interior grasping surface or at an internal location within thefingers 540. The force sensors can include flat and/or compact forcesensors such as force-sensitive resistors. In some embodiments, the oneor more force sensors are placed at the interface between the resilienttip and an inward-facing portion of the base 542, such that the forceexerted between an inner surface of the tip 544 and an item in thetarget object space is transferred to and measured at the force sensor.In some embodiments, the force sensors may be attached to theinward-facing portion of the base 542 such that the resilient tips maybe removed and/or replaced while the force sensors remain attached tothe fingers 540. Some embodiments may include more than one force sensorin each finger 540, for example, to provide redundancy and/or to providemore detailed data regarding the force being applied to an object by thefingers. Accordingly, an increase in the force measured at the fingerforce sensors relative to a baseline may indicate to the controller 501or other processing circuitry that a finger 540 has contacted an item.

The forces detected at the finger force sensors may further be used tocontrol the linear actuators 535. For example, a predetermined thresholdforce value may be associated with a maximum grasping force to beapplied to items (e.g., to avoid bruising fruit being picked by the endeffector 500). In this example, the linear actuators 535 may stop distalmovement of the carriage 530 when the force detected at any of thefinger force detectors reaches or exceeds the predetermined thresholdvalue. As will be described in greater detail below, differences betweenforces detected at individual fingers 540 may further be used todetermine a direction from which the grasped item is tethered to anotherobject (for example, the direction of a fruit stem tethered to a plant).

The end effector may further include one or more cameras 550 or otherimaging devices. The cameras 550 include photodetectors configured toimage an area below and/or laterally displaced from the end effector. Insome embodiments, image data from two or more cameras 550 may be usedfor stereo imaging or other techniques for determining depth informationof the imaged region. Image data obtained at the cameras 550 may beprocessed to identify imaged items to be picked, as well as to determinelocation information associated with the items. In some embodiments,some or all of the cameras 550 may further include one or more lightsources at or near the cameras 550 to illuminate the cameras' field ofview, for example, to provide controllable illumination which mayimprove the reliability of item detection and identification. In someembodiments, the light sources may provide illumination from a sourcethat is functionally coincident or nearly coincident with the camera orcameras. Thus, as the cameras and lights get closer to an object, thelight intensity reflected off the object and thus present on thecamera's imager increases. In some cases, such increased intensity mayundesirably interfere with accurate imaging. In some aspects, theintensity of the lighting may be controlled as a function of distancefrom an item being approached, such that light is emitted with a lowerintensity when the item of interest is close to the light source andcamera. In other aspects, the intensity of the lighting may becontrolled as a function of the intensity of light detected at thecamera.

Example Robot-Level Harvesting Process According to the PresentDisclosure

With reference to FIGS. 6A-6G, an example picking sequence using theexample end effector 500 of FIGS. 5A-5D will now be described. Thepicking sequence may be performed under control of one or more computingdevices and/or control circuitry of the harvesters, work cells, robots,and end effectors described herein. For example, the picking sequencemay be controlled at least in part by one or more software processesexecuting on some or all of the vehicle master controller 402, slavecontroller 426, end effector hub 428, controller 440, or otherprocessing components illustrated in FIGS. 4A and 4B. Although theexample picking sequence of FIGS. 6A-6G is depicted and described withreference to the particular agricultural implementation of picking astrawberry from a strawberry plant, it will be understood that thissequence or similar sequences may equally be used for other agriculturalor non-agricultural picking implementations.

As described above with reference to FIGS. 5A-5E, the end effector 500includes, among other components, a shaft 520, a carriage 530, a suctiondevice such as a suction cup 522, and a plurality of fingers 540. Theshaft 520 extends along the longitudinal axis 521 between a proximal end527 _(P) and a distal end 527 _(D). The carriage 530 is configured torotate about and translate along an intermediate portion 527 _(I) of theshaft 520 between a proximal position and a distal position. The suctioncup 522 is coupled to the distal end of the shaft and forms thedistal-most portion of the end effector 500 when the carriage 530 is inthe proximal position. The fingers 540 are spaced radially about thecarriage 530, with each finger 540 coupled to the carriage by a hinge athinge point 536 and extending distally from the carriage 530. Theplurality of fingers 540 envelop the intermediate portion 527 _(I) ofthe shaft 520 when the carriage 530 is in the proximal position. Whenthe carriage 530 is in the distal position, the plurality of fingers 540envelop a target object space including the distal end 527 _(D) of theshaft and the suction cup 522. The fingers form the distal-most portionof the end effector 500 when the carriage 530 is in the distal position.

The picking sequence begins with the end effector 500 in the initialconfiguration depicted in FIG. 6A. In the initial configuration of FIG.6A, the carriage 530 and fingers 540 are in a retracted configuration,and are positioned at or near the proximal end of the shaft 520. In thisinitial configuration, the portion of the end effector 500 distal to thecarriage 530 has a relatively high aspect ratio (that is, is longeralong the longitudinal axis relative to the width of this portionperpendicular to the longitudinal axis) because the shaft 520 andsuction cup 522 extend significantly beyond the ends of the fingers 540.In accordance with the item identification and selection methodsdescribed below, the robot corresponding to the end effector 500 hasmoved the end effector 500, generally along a downward direction 602, toa position at which the suction cup 522 engages a strawberry 65 attachedto a stem 70. In this initial configuration, the strawberry 65 coverssubstantially all of the opening 523 of the suction cup 522 andtherefore experiences an upward suction force along the longitudinalaxis. The upward suction force causes the suction cup 522 to suctiongrasp the strawberry 65. This engaging of the strawberry at the suctioncup 522 may be detected based on data from a vacuum sensor, a linearposition sensor, and/or a force sensor as described with reference toFIGS. 5A-5D. When the strawberry 65 has been engaged by the suction cup522, in some embodiments the end effector 500 moves longitudinallyupward, opposite the direction 602, to pull the strawberry 65 clear ofstems, leaves, dirt, or other clutter or debris that may have beenlocated near the strawberry 65.

Referring now to FIG. 6B, after the strawberry 65 has been initiallygrasped by suction, the carriage 530 and fingers 540 advance to anextended configuration. In this example, the linear actuators 535 movethe carriage 530 and fingers 540 by extending the rods 534 in a distaldirection 604, causing the carriage slider 532 to move along thelongitudinal axis toward the distal end of the shaft 520. FIG. 6Bdepicts the intermediate position described above at which the fingerextensions 546 contact the collar 526 of the shaft 520 after the slider532 has moved longitudinally in a distal direction. At the positiondepicted in FIG. 6B, the volume of the target object area enveloped bythe fingers 540 has not changed and remains at its largest extent.

Referring now to FIG. 6C, the linear actuators 535 continue moving therods 534, carriage slider 532, carriage 530, and fingers 540longitudinally along the distal direction 604. As the carriage 530continues beyond the intermediate position of FIG. 6B, the fingers 540rotate about the hinge points 536 of the carriage 530 due to the fingerextensions 546 being constrained by the collar 526. Thus, the fingers540 close about the strawberry 65 to form a positive mechanical grasp.The linear actuators 535 may stop moving the rods 534 in the distaldirection, and thus stop tightening the fingers' 540 grasp on thestrawberry 65, in response to a threshold force detection at one or moreforce sensors within the fingers 540.

With reference to FIG. 6D, after the fingers 540 have positively graspedthe strawberry 65, the robot moves the end effector 500 longitudinallyupward along a proximal direction 606. As the end effector 500 movesupward, the strawberry 65 moves further from the base of its stem 70,thereby creating tension in the stem 70. As the stem 70 begins totighten, the forces detected in the fingers' force detectors may change.For example, because the tension in the stem 70 tends to pull thestrawberry 65 back toward the stem 70, the force sensors in the twofingers 540 adjacent to the stem 70 detect an increase in force, whilethe force sensor in the finger 540 opposite the stem 70 detects adecrease in force. In addition, the relative increases in force detectedat the two stem-adjacent fingers may be used to more precisely determinethe direction of the stem 70 relative to the end effector.

With reference to FIG. 6E, after the direction of the stem 70 has beendetermined based on changes in force determined during vertical motionof the strawberry 65, the picking sequence continues to an optionaluntensioning step. During the untensioning step, the robot moves the endeffector 500 along a curved path 608. Using the anchored stem 70 as aradius, the end effector 500 can trace a portion of a circular arcdefined by the length of the stem 70, thus moving the strawberry 65 to aposition closer to the point directly above the base of the stem. Movingthe strawberry 65 upward and toward the base of the stem at the plant,can advantageously reduce the amount of force necessary to separate thestrawberry 65 from the stem 70. This reduction in force can enhance theability of harvesters described herein to pick and place soft-bodied ordelicate objects, such as berries, with little or no damage, while alsomaintaining very high picking efficiency.

After the optional untensioning step, the picking sequence continues tothe configuration of FIG. 6F. In FIG. 6F, the shaft 520, suction cup522, carriage 530, and fingers 540 have rotated in a rotation direction610 about the longitudinal axis to rotate the strawberry 65. Thestrawberry 65 may be rotated by 45 degrees, 60 degrees, 90 degrees, 120degrees, 180 degrees, or more. In various embodiments, the rotationangle may be pre-determined or may not be pre-determined. For example,if the shaft actuator 515 is a pneumatic rotary actuator, the shaft 520may only be rotatable to two positions; if the shaft actuator 515 is aservomotor, the shaft 520 may be rotatable to a larger number ofselectable positions. Rotation of the berry 65 about the longitudinalaxis may advantageously further reduce the amount of force necessary toseparate the strawberry 65 from the stem 70 (e.g., in some cases a stemthat requires a force of approximately 20N to separate without rotationmay only require approximately 2N if the berry has been rotated).Although the rotation direction 610 is depicted as beingcounterclockwise from a top-down perspective, it will be understood thatthe shaft 520, suction cup 522, carriage 530, and fingers 540 may alsorotate clockwise, and an individual end effector 500 may be capable ofboth clockwise and counterclockwise rotation. For example, in someembodiments an end effector 500 may rotate a berry in a first rotationdirection the first time it attempts to pick a berry, and, following afailed pick, may rotate the berry in the opposite rotation directionwhen making a second attempt to pick the berry. After the strawberry 65has been untensioned and rotated, it is ready to be removed from thestem 70.

Referring now to FIG. 6G, the picking sequence is completed as the robotmoves the end effector 500 upward along direction 612, separating thestrawberry 65 from the stem 70 (not visible in FIG. 6G). In order tobreak the stem 70 and/or cause the stem 70 to detach from the calyx 67of the strawberry 65, the end effector 500 may move upward at a highspeed and/or acceleration. In some embodiments, the robot may move theend effector 500 upward, with the fingers 540 still positively graspingthe strawberry 65, at an acceleration in the range of 3 g to 5 g ormore, and/or exerting a stem separation force of 2N or more. During thisrapid acceleration, the resilient and/or cushioned material forming thetips 544 of the fingers 540 may cushion the strawberry 65 to prevent itfrom being bruised during separation from the stem 70. In addition, theuntensioning and rotation performed prior to separating the strawberry65 from the stem 70 may further reduce the amount of force necessary tobreak the stem 70, thereby reducing the force that must be exerted onthe strawberry 65 by the fingers 540 during this step. When the pickingsequence has been completed, the strawberry 65 may be placed onto aconveyor and/or into a receptacle for storage, such as by positioningthe end effector 500 above the receptacle and moving the carriage 530and fingers 540 back to a retracted configuration at the proximal end ofthe shaft 520. Suction at the suction cup 522 may also be interrupted,and/or the end effector may be moved in a jerking motion, to break thegrasp at the suction cup 522 and cause the strawberry 65 to fall intothe receptacle.

Example Harvester-Level Harvesting Process According to the PresentDisclosure

FIG. 7 is a flow chart illustrating an example harvester-level method700 for picking a berry according to the present disclosure. The method700 may be performed by a computer system integrated within a systemsuch as all or a subset of the processing components of the harvesters100, 300, 400 depicted in FIGS. 1-4B. Throughout the description ofmethod 700, reference will be made to example components of FIGS. 4A and4B, but it will be understood that the method 700 can be implemented inany harvester according to the present disclosure.

The method 700 begins with block 702, where the harvesting process isinitiated. The method 700 may begin, for example, based on a manualcommand to begin harvesting provided at a user interface 414 of aharvester 400. Alternatively, the method 700 may be initiatedautonomously based on a determination by the vehicle master controller402 that the harvester 400 is in a location where items will be picked(for example, based on detection of items to be picked, predeterminedlocation or geofencing, etc.). When the harvesting process has beeninitiated, the method 700 continues to block 704.

At block 704, the work cell 416 _(N) idles. For example, one or moreprocessing components, controllers, or other devices may need to executeone or more tasks prior to harvesting. These tasks can be executedduring the idle state. When the work cell 416 _(N) has completed tasksduring the idle state, the method 700 may continue to block 706 in whicha robot check is performed. During the robot check, the robot controller440 or other component of the robot 422 ₂ may verify that sensing andcontrol components of the robot 422 ₂ (such as sensors 434, 436, 438 andservos 424) are in communication with the controller 440 and are workingproperly. If an error is detected during the robot check, the methodcontinues to block 708 and resets one or more systems to return to idleat block 704. If an error is not detected at block 706, the methodcontinues to block 710.

At block 710, one or more cameras 430 ₁, 430 ₂, 430 ₃ in the work cellbegin acquiring image data of the picking area corresponding to the workcell. For example, the cameras may begin imaging a row of strawberryplants disposed below the harvester in the work volume. Images may beobtained during a discrete image-gathering phase, and/or may be obtainedreal-time during harvesting. For example, with cameras mounted on theend effectors described herein, image data from multiple locations maybe obtained throughout the picking process. When the cameras have begunacquiring image data, the method 700 continues to block 712.

At block 712, the harvester scans (e.g., by causing the robot to move anend effector containing one or more cameras) to detect items (e.g.,berries) to be picked. Items to be picked may be detected and/oridentified based on various computer vision methods and/or any otherimaging and image processing methods. In one example implementation ofstrawberry harvesting, color images of the picking area obtained fromthe cameras 430 ₁, 430 ₂, 430 ₃ are processed. Color images taken at adownward angle of a row containing strawberry plants may include, forexample, brown areas corresponding to soil, green areas corresponding toplant leaves, stems, and unripe strawberries, white areas correspondingto flower petals and/or unripe strawberries, and red areas correspondingto ripening, ripe, or overripe strawberries for harvesting. Regions ofone or more red pixels may thus be identified from the color images aspossible berry candidates. The identified red regions may further beanalyzed to determine the size, shape, and/or hue of each identified redregion. Other suitable scanning techniques to detect different kinds ofitems of interest can be implemented.

Berry candidates may additionally be tracked from frame to frame for amoving camera (e.g., using an Intersection over Union metric or othersuitable method to correlate identified regions from frame to frame).Potential berry candidates may thus be identified based on data frommultiple cameras at multiple locations and converted to a globalcoordinate system for use by all robots in the work cell or for use bythe robot that acquired the image at a later, more optimal time. When atleast one berry candidate has been identified, the method 700 continuesto decision state 714.

At decision state 714, the controller 440 determines whether a berry hasbeen identified with a high confidence level. For example, one or moreberry candidates may be analyzed to determine, for example, if thenumber of red pixels is large enough to correspond to a strawberry witha high level of confidence. In another example, the confidence level maybe based at least in part on the shape of the region defined by the redpixels. In another example, a candidate berry may also be analyzed todetermine if the hue of the pixels is a dark enough red to be confidentthat the berry is ripe enough to be picked. Berry candidate analysis maybe implemented with any of various known statistical methods and/orthresholding processes, including neural networks, etc.

In some embodiments, the controller 440 may receive informationregarding identified berry candidates from another robot within the samework cell. Sharing of information regarding berry candidates betweennearby robots may be advantageous, especially in eye-in-handimplementations in which the cameras are located on and move with therobots and/or end effectors, by allowing robots to be informed of berrycandidates that are accessible to their respective end effector but notvisible to their respective cameras due to clutter in the environment(for example, a berry blocked from view by leaves but visible to asecond robot nearby).

If it is determined at decision state 714 that a berry candidate cannotbe confirmed with a sufficiently high confidence level, the method 700continues to block 716 to search for another berry candidate in theimage data. If no berry is found, the method 700 returns to block 712 tocontinue scanning for additional berry candidates in the picking area.If it is determined at decision state 714 that the berry candidate canbe confirmed with sufficient confidence, or if another berry candidateis confirmed at block 716, the method 700 continues to block 718.

At block 718, the work cell reaches the location associated with theconfirmed berry candidate and, at block 720, the robot initiates theprocess of attempting to pick the berry. Blocks 718 and 720 may includeone or more of moving the harvester, moving a robot within the harvesterto an appropriate position, and preconfiguring the end effector to beginthe pick sequence. At block 722, the robot and end effector execute arobot-level pick sequence to grasp the berry and separate it from itscorresponding plant. An example robot-level harvesting process isdescribed above with reference to FIGS. 6A-6G. It will be understood,however, that the robot-level pick sequence may be any suitable picksequence corresponding to the particular gripping mechanism being used,and is not to be interpreted as limited to the gripping mechanismsdescribed herein. Accordingly, the work-cell level harvesting processmay include the sequences depicted in FIGS. 6A-6G and 8 but is notlimited to these sequences or the particular gripping mechanismsdescribed in these figures. After the pick sequence is completed, themethod 700 continues to decision state 724.

At decision state 724, the controller 440 or other processing componentdetermines whether the pick was successful. In some embodiments, thedetermination at decision state 724 may be made after the robot attemptsto break the berry stem and before moving the berry to the receptacle.At decision state 724, the controller 440 may analyze measurementsreceived from the vacuum sensor 434 and/or finger force sensors 438. Forexample, if the force detected at the finger force sensors decreases toa baseline reading, or if the vacuum sensor no longer detects a lowpressure, it may be determined that the berry disengaged from (or wasdropped by) the end effector during the pick sequence, and that theattempted pick was not a success. If the finger force sensors continueto detect that force is being applied between the fingers and an item,or if the vacuum sensor still detects that an item is blocking thesuction cup opening, it may be determined that the berry is still in thegripper, and that the pick was a success. If it is determined atdecision state 724 that the pick was not successful, the methodcontinues to block 726 where a reassessment occurs and the end effectordoes not continue moving toward the receptacle. Based on thereassessment, the method 700 may either return to block 720 to reattemptto pick the berry or may return to block 712 to scan for additionalberry candidates or receive information from images taken by anotherrobot that an identified berry is a candidate for picking.

If it is determined at decision state 724 that the pick was a success,the method 700 terminates at block 728, where the berry is dropped intothe receptacle. The process of dropping the berry may include moving theend effector toward the receptacle but releasing the berry before theend effector is above the receptacle (thereby relying on the velocityand trajectory of the berry to carry it to the receptacle), or it caninclude moving the end effector to a position directly above thereceptacle. For example, the robot may throw or toss the berry into thereceptacle before the end effector is located physically above thereceptacle. The point at which the berry is released can be determinedby balancing the desire not to damage the berry when it lands in thereceptacle and the desire to re-allocate the end effector to the nextpicking sequence as quickly as possible, increasing efficiency of thepicking operation. Once the berry has been dropped into the receptacle,the picking process is complete and the work cell may continue pickingadditional berries. If continued operation is desired, the method 700returns to block 712 to scan for additional berry candidates.Alternatively, if another berry candidate has already been identifiedwith sufficient confidence, the method 700 may return directly to block718 to begin picking the next berry candidate without an additionalscanning operation. Blocks 712-728 may be repeated indefinitely duringoperation of a harvester according to the present disclosure.

Example End Effector-Level Picking Sequence According to the PresentDisclosure

FIG. 8 is a flow chart illustrating an example end effector-level method800 for picking a berry according to the present disclosure. The method800 may be performed by a computer system integrated within a systemsuch as all or a subset of the processing components of the harvesters100, 300, 400 depicted in FIGS. 1-4B. For example, the method 800 may beperformed by the end effector 500 of FIGS. 5A-6G, moved by a robot 422 ₂and under control of the controller 440 or a slave controller 426.Although the method 800 will be described herein with reference to theend effector 500 of FIGS. 5A-6G, the method 800 may equally beimplemented with any other suitable end effector.

The method 800 begins at block 802 when the pick sequence is initiated.The pick sequence may be initiated automatically, for example, when awork cell including the robot 422 ₂ and end effector 500 detect a berrywith a high level of confidence and enter a pick sequence at decisionstate 714 and/or block 720 of FIG. 7. Alternatively, the pick sequencemay be initiated manually based on an input received at a user interface414. When the pick sequence has been initiated, the method 800 continuesto block 804.

At block 804, the controller 440 causes the robot 422 ₂ to move the endeffector 500 along a ray approach path to the berry to be picked. Theray approach path may be a predetermined ray approach path calculated atthe controller 440 or other processing component of the harvester 400based on the location of the berry as determined during the berrycandidate identification and selection processes described above withreference to FIG. 7. In some embodiments, the ray approach may becalculated so as to be coincident with a line of sight along which theberry was viewed and identified during the berry candidate selectionprocess. Advantageously, approaching the berry along a known line ofsight (for example, a path known to be clear of obstructions) mayprovide a high probability that the suction cup 522 will be able toreach the berry without being blocked by leaves, stems, or other debristhat may prevent the suction cup 522 from reaching or successfullyengaging the berry. After the ray approach path to the berry has begun,the method 800 continues to block 806.

At block 806, one or more sensors of the end effector 500 can bemonitored along the ray approach to detect when the suction cup 522 hasreached the berry. For example, the ray approach may be calculated as aray extending to and through the expected berry location. As describedabove, approaching the berry along a ray that extends through theexpected location can mitigate the effects of berry movement. In astrawberry field, individual berries may move frequently. For example,an effector picking a nearby strawberry might move a target berry orrustle a stem or leaves that cause the target berry to move. Individualberries may also move due to harvester vibration, wind, or other forces.This movement may be slight, and moving through the initial position maystill result in a successful contact with a target berry despite somemovement. While the end effector 500 travels along the ray approach, thevacuum sensor and/or linear position sensor may be monitoredcontinuously or periodically to detect formation of a vacuum at thesuction cup 522, activation of a touch switch, and/or linear motion ofthe suction cup 522 upward along the longitudinal axis caused by runninginto the berry. When it is detected that the suction cup 522 has reachedand engaged the berry (or an item estimated to be a berry with a highdegree of confidence), the method 800 continues to block 810 or block814.

In situations when engagement was detected at block 806 due toactivation of both the vacuum sensor and the touch sensor, the method800 can continue to block 810, in which the robot moves the end effector500 upward to lift the berry away from the plant (for example, by 1 cm,2 cm, 5 cm, or more) until the touch sensor disengages, before vacuumlifting at block 814 and extending the fingers 540 to grasp the berry atblock 816. However, if engagement was detected based only on activationof the touch sensor and a vacuum was not also created at the suctioncup, it is possible that the berry has been found by the suction cup 522but not successfully vacuum grasped. In this situation, the method 800may proceed directly from block 806 to block 812, as the suction cup 522may not have a sufficient grasp on the berry to lift it. In contrast, ifengagement was detected at block 806 due to activation of the vacuumsensor instead of or in addition to activation of the touch sensor, themethod 800 continues from block 806 to lift and positively grasp theberry at blocks 814 and 816. When the fingers 540 have been extended topositively grasp the berry at block 812 or block 816, the method 800continues to block 818.

At block 818, the robot lifts the end effector 500 upward in the z-axisdirection such that the berry is lifted further upward along thelongitudinal axis of the end effector shaft. If the berry is stillconnected to its stem, the stem will become more taut as the berry islifted upward while being grasped by the fingers 540. As tension in thestem increases, the berry experiences an increasing force pulling alongthe direction of the stem. Accordingly, the force sensors in the twofingers 540 adjacent to the stem direction will detect an increasedforce, while the remaining finger 540 will detect a decreased force.When these varying forces are detected, the controller 440 may determinethat the azimuth corresponding to the actual stem direction is somewherebetween the azimuths of the two fingers 540 experiencing the increasedforce. The relative force increases at the two stem-adjacent fingers 540may further be used to more accurately determine the stem azimuth. Forexample, if the two force increases are approximately equal, it may bedetermined that the stem originates from an azimuthal directionapproximately equally between the two fingers 540. If one of the twostem-adjacent fingers 540 experiences a higher increased force than theother stem-adjacent finger 540, it may be determined that the stemazimuth is closer to the finger 540 sensing the higher increased force.In various embodiments, the actual stem direction may be determined withrelatively high accuracy using known trigonometric operations, subjectto the sensitivity and/or resolution of the force sensors in the fingers540. After the berry has been lifted and the stem direction has beendetermined, the method 800 continues to block 820.

At block 820, the stem is untensioned to facilitate further steps in themethod 800. For example, reducing tension in the stem may ensure thatthere is enough slack to rotate the berry. Based on the known stemdirection, the robot moves the end effector 500 upward and laterallytoward the base of the stem. In some embodiments, the end effector 500may roughly trace a circular arc constrained by the stem as a radius.The end effector 500 may continue moving until it approximately reachesa zenith at which the stem is generally oriented along the longitudinalaxis of the shaft, or may stop before reaching the zenith. Thus,following the untensioning step, the stem is more closely aligned withthe longitudinal axis of the end effector relative to the initialgrasping location, while the portion of the stem near the calyx is stillhorizontal. Thus, a subsequent pulling force along the longitudinal axiswill result in a high moment applied at the point where the stem entersthe berry, advantageously reducing the amount of force required toseparate the berry from the stem by motion along the longitudinal axis.After the stem has been untensioned, the method 800 continues to block822.

At block 822, the berry is rotated about the longitudinal axis of theend effector shaft. In the example end effector 500, the shaft actuator515 causes the shaft 502 to rotate about the longitudinal axis 521 suchthat the suction cup 502, carriage 530, fingers 540, and berry allrotate about the longitudinal axis 521. The degree of rotation may be,for example, in the range of 45 degrees, 60 degrees, 90 degrees, 120degrees, 180 degrees, or more. In some implementations, the rotationstep may be skipped and the method 800 may proceed directly from block820 to block 824. For example, some plant varieties may have weakerstems than other varieties, reducing the need to include a rotationstep. In another example, plant stems may grow longer and become easierto break over the course of a season. Thus, the rotation step may beincluded early in a strawberry harvesting season while the stems areshorter and stronger, and may be omitted later in the season when thestems are longer and weaker. After the berry has been rotated, or if therotation step is not being performed, the method 800 continues to block824.

At block 824, the robot pulls the end effector 500 upward (generally inthe z-axis direction) to pull the berry away from the stem and removethe berry from the plant. The pulling motion may be a sudden upwardmotion with acceleration in the range of 3 g-5 g or more to providesufficient force to cause the stem to break or to separate from theberry at the calyx. If the stem is broken or separated from the berry(e.g., as determined by monitoring the finger force sensors, vacuumsensors, and/or camera sensors), the method 800 terminates at block 826when a successful pick is determined.

LIDAR Guidance

With reference to FIGS. 9A-10, autonomous harvesters described hereinmay be directed by guidance systems using LIDAR or the like. Forexample, the guidance system may use LIDAR to detect one or more furrowsand/or beds of an agricultural field and identify a path or otherwiseguide the harvester along the furrows without damaging the beds locatedbetween the furrows, as described above with reference to FIG. 3.

FIG. 9A is a top-down schematic view depicting a harvester 300positioned over a furrow 50 and a bed 55 of an agricultural field, suchas a strawberry field. The harvester 300 includes wheels 305 and aguidance sensor 330, which in this example is a detector of a LIDARsystem. The wheels 305 preferably travel along the bottom of the furrow50 near a furrow centerline 80, such that the intermediate portion ofthe harvester 300 is suspended above the bed 55 and does not contact thebed 55 or items such as plants located on the bed 55.

The guidance sensor 330 may be mounted on a front portion of theharvester 300. For example, the guidance sensor 330 may be located aheadof other components of the harvester. The guidance sensor 330 can beoriented to point directly downward or at a downward angle so as todetect physical features of the furrow 50. The guidance sensor 330 ismounted in-line with a wheel centerline 82 or at a known lateral offsetrelative to the wheel centerline 82. The wheel centerline 82 may becoincident with the furrow centerline 80 or may be linearly offset by awheel offset distance 84 along the x-axis at a location near theguidance sensor 330. The wheel centerline 82 may further be angularlyoffset from the furrow centerline 80 by an offset angle 86. The guidancesensor 330 can be mounted at a known baseline distance 88 along they-axis relative to a central location 312, such as a center of rotationof the harvester 300. In some embodiments, the offset angle 86 may becalculated trigonometrically based on the wheel offset distance 84 andthe baseline distance 88.

Where the guidance sensor 330 is a LIDAR device, the guidance sensor 330can be a line scanning device configured to determine radial distancesfrom the sensor to objects or portions of objects within the field ofview of the guidance sensor 330. Locations of objects or portions ofobjects may be readily determined in polar coordinates, with theguidance sensor 330 at the origin, based on the distance and angle tothe object or portion thereof, as detected by the guidance sensor 330.

FIG. 9B illustrates an example LIDAR implementation in a furrow 50. Thefurrow 50 is located adjacent to a bed 55. Furrow walls 52 extendbetween the base 54 of the furrow and the bed 55. The furrow 50 can begenerally defined by a base width BW, a top width TW, and a furrowheight FH. The furrow walls 52 may be vertical such that TW is equal toBW, or may be disposed at an angle such that TW is greater than BW. Theguidance sensor 330 (e.g., a LIDAR system or sensor thereof) is disposedat a LIDAR height LH above the base 54 of the furrow 50. In thisposition, the guidance sensor 330 may detect various features of thefurrow 50. For example, the guidance sensor 330 may detect a furrow baseangle ABF between a sensor centerline 331 and an intersection betweenthe base 54 and a wall 52, a furrow top angle ATF between the sensorcenterline 331 and an intersection between a bed 55 and a wall 52, and atop-to-top angle ATT between the tops of the two furrow walls 52.

Referring jointly to FIGS. 9A and 9B, the LIDAR guidance system can usethe angles and distances to various points within the furrow 50 todetect the locations of the furrow walls 52. Based on the locations ofthe furrow walls, the guidance system can determine the location of thefurrow centerline 80 and the corresponding wheel offset distance 84.Based on the wheel offset distance 84, the guidance system can cause theharvester 300 to adjust its course (e.g., by communicating with avehicle master controller or other vehicle control component). Theharvester 300 may be guided such that the wheels 305 are positioned atthe furrow centerline 80 or within a desired distance from the furrowcenterline 80 (e.g., the wheel offset distance 84 is 0 or close to zero,such as within 1 inch, 2 inches, inches, 6 inches, 1 foot, 1.5 feet,etc.).

Referring now to FIG. 10, an example LIDAR guidance method 1000 will bedescribed. As shown in FIG. 10, the method 1000 may be a repeated and/oriterative process performed continuously or periodically as a harvestermoves along a furrow at block 1005. As the harvester moves along thefurrow or before the harvester begins moving, the LIDAR system obtainsdata at block 1010. The LIDAR data may include a set of data points inpolar coordinates, with each point comprising an angle and a distancecorresponding to a point along the bed 55, furrow wall 52, or furrowbase 54 as depicted in FIG. 9B.

At block 1015, the LIDAR data may be trimmed to focus on a region ofinterest. For example, the data may be trimmed to exclude points atangles outside the angle ATT depicted in FIG. 9B. The data points withinthe LIDAR data may additionally be converted from polar coordinates toCartesian coordinates (e.g., within the x-z coordinate plane depicted inFIG. 9B). At block 1020, the Cartesian converted LIDAR data may beconvolved with a kernel (e.g., a mask or convolution matrix) to producean output signal. For example, an edge detection kernel or similarkernel may be used to identify peaks corresponding to the edges of thefurrow walls. In one particular example, the kernel may include a firsthalf having values of −1 and a second half having values of +1, suchthat the output signal has positive peaks corresponding to the fallingedges of the furrow walls 52 and negative peaks corresponding to therising edges of the furrow walls 52.

At block 1025, the peaks in the output signal are detected, and theirlocations along the x-axis (FIG. 9B) are determined relative to thesensor centerline 331. Based on the locations of the peaks, at block1030 the system calculates the wheel offset distance 84 as shown in FIG.9A. At block 1035, the offset angle 86 is calculated based on the offsetdistance 84 and the known baseline distance 88. The offset angle 86 maythen be compared with zero as a negative feedback for the vehiclecontrol system. At block 1040, the motion of the harvester may bedetermined in Cartesian coordinates based on the computed offset angleand a speed set point 1045 corresponding to a linear speed of theharvester along the offset angle direction. At block 1050, a command maybe issued to the vehicle motor controller, for example, to change thedirection of motion or to continue along the same path. For example, ifit is determined during the method 1000 that the offset angle is 0 andthe offset distance is 0 or close to 0, the guidance system may directthe vehicle motor controller to continue along the current path withoutadjustment. However, if it is determined that the offset angle is not 0,the guidance system may direct the vehicle motor controller to changethe direction of motion (e.g., by steering one or more wheels or bychanging the relative rotational velocities of the left-side andright-side wheels) to reduce the offset angle. After the method 1000 iscomplete at block 1050, the method 1000 can repeat indefinitely as theharvester continues moving along the furrow at block 1005.

Additional End Effector Details

FIGS. 11A-11D depict additional views of the end effector 500 of FIGS.5A-5E during the sequence of FIGS. 6A-6G. Specifically, FIG. 11Acorresponds to the configuration of FIG. 6A, FIG. 11B corresponds to theconfiguration of FIG. 6B, FIG. 11C corresponds to the configuration ofFIG. 6C, and FIG. 11D corresponds to the configuration of FIG. 6F.

Example Robot According to the Present Disclosure

Harvesters described herein can advantageously implement any suitablerobot 422 ₁, 422 ₂ 422 _(N) in accordance with the present disclosure.In addition to the example delta robot implementation described abovewith reference to FIG. 3A, a second example robot implementationillustrated in FIG. 3B will now be described. It will be understood thatthe present technology is not limited to either of the example robotsdescribed herein, and other suitable robots, or variations of the robotsdescribed herein, can be implemented.

FIGS. 12A-12G depict an example robot 1200 that may be used to mount andmove any of the end effectors described herein. The example robot 1200is illustrated as a t-type robot 1200 configured to support an endeffector 1210 and to move the end effector 1210 as desired within aharvester work cell or other picking area. With reference to FIG. 12A,the robot 1200 generally includes a radial member 1220, a carriage 1230,a longitudinal member 1240, and mounting brackets 1250 disposed atopposite ends of the longitudinal member 1240. The longitudinal member1240 may be a gantry or other generally linear component.

The end effector 1210 is mounted to a distal end 1224 of the radialmember 1220 at a mounting plate 1212. The mounting plate 1212 is coupledto an end effector actuator gear 1214 allowing the end effector 1210 tobe rotated relative to the radial member 1220. The radial member 1220,which may be an arm, tube, or the like, includes radial member tracks1222 retained by radial member rollers 1232 of the carriage 1230, suchthat the radial member 1220 is retained by the carriage 1230 but canslide relative to the carriage 1230 along a direction parallel to thelength of the radial member 1220. The carriage 1230 further includeslongitudinal member rollers 1236 counted on an opposite side of a plateof the carriage relative to the radial member rollers 1232. Thelongitudinal member rollers 1236 retain longitudinal member rails 1242of the longitudinal member 1240 such that the carriage 1230 remainsattached to the longitudinal member 1240 but can slide relative to thelongitudinal member 1240 along a direction parallel to the length of thelongitudinal member 1240. The longitudinal member 1240 is coupled to themounting brackets 1250 by longitudinal member bearings 1252 (e.g., slewrings or other rotational bearings), such that the longitudinal member1240 is fixed to the mounting brackets 1250 but can rotate about an axisparallel to the length of the longitudinal member 1240. In someembodiments, the longitudinal member 1240 may include a motor (notshown) which may be mounted inside the longitudinal member 1240 andconfigured to rotate the longitudinal member 1240 about the axis.

Movement of the carriage 1230 along the longitudinal member 1240 andmovement of the radial member 1220 relative to the carriage 1230 may becontrolled by one or more motors, such as motors 1248 a and 1248 b. Insome embodiments, two or more motors may individually control movementof the radial member 1220 and the carriage 1230. In some embodiments,such as in the embodiment depicted in FIGS. 12A-12G, motors 1248 a, 1248b may operate together to jointly control motion of both the radialmember 1220 and the carriage 1230. Motors 1248 a and 1248 b includeoutput shafts 1246 a, 1246 b shaped to engage and drive one or morebelts (not shown) that couple to the longitudinal member 1240, thecarriage 1230, and/or the radial member 1220 around idler pulleys 1244a, 1244 b, 1245 a, 1245 b, 1238 a, 1238 b, 1222, 1239 a, and 1239 b. Inone example configuration, a single belt is coupled at a first end tothe radial member 1220 at a first belt securement 1228 a and is looped,in order, over idler pulleys 1239 a and 1244 a, output shaft 1246 a,idler pulleys 1245 a, 1238 a, 1222, 1238 b, and 1245 b, output shaft1246 b, and idler pulleys 1244 b and 1239 b. A second end of the belt iscoupled to the radial member 1220 at a second belt securement 1228 b.

In one example, motors 1248 a and 1248 b are utilized in a CoreXYcontrol arrangement. In the CoreXY arrangement, a belt (not shown) iscoupled to the components of the robot 1200 such that simultaneousmotion of both motors 1248 a, 1248 b can produce translation of theradial member 1220, translation of the carriage 1230, or both. Forexample, simultaneous motion of the motors 1248 a, 1248 b in the samedirection at the same speed may cause the radial member 1220 to moverelative to the carriage 1230 while the carriage 1230 remainsstationary, and simultaneous motion of the motors 1248 a, 1248 b inopposite directions at the same speed may cause the radial member 1220to remain stationary with respect to the carriage 1230 while thecarriage 1230 moves along the longitudinal member 1240. Combined motionof the radial member 1220 and the carriage 1230 may be achieved bydriving the two motors 1248 a, 1248 b at different speeds and/or indifferent directions simultaneously. Additional details of CoreXYcontrol arrangements are generally known in the art and will not beprovided here. It will be understood that the term “CoreXY” does notnecessarily indicate that the motors 1248 a and 1248 b are limited tocontrolling motion within a particular (x,y) coordinate system, but areconfigured to control the motion along substantially perpendiculardirections parallel to the longitudinal member 1240 and the radialmember 1220, regardless of the rotational orientation of thelongitudinal member 1240 with respect to the mounting brackets 1250. Itwill further be understood that other suitable control arrangements tocontrol movement of the carriage 1230 and the radial member 1220 couldbe implemented in accordance with the present disclosure.

FIGS. 12B-12D illustrate the various axes of motion achievable with theexample robot 1200. Referring jointly to FIGS. 12B-12D, the end effector1210 may be translated along a longitudinal axis 1203 parallel to thelongitudinal member 1240, as shown by motion 1202. Motion of the endeffector 1210 along the longitudinal axis 1203 is achieved by moving thecarriage 1230 along the longitudinal member 1240 as described above withreference to FIG. 12A. The end effector 1210 may further be translatedtoward or away from the longitudinal member 1240 along a radial axis1205 parallel to the radial member 1220, as shown by motion 1204. Whenthe longitudinal member 1240 is rotated such that the radial member 1220is vertical, motion 1204 along the radial axis 1205 corresponds tovertical up-and-down motion.

The radial axis 1205 may be rotated relative to the longitudinal member1240 about a first rotational axis 1207 a as represented by rotationalmotion 1206 a. The longitudinal member 1240 extends into the page ofFIG. 12C, but a portion of the end of the longitudinal member 1240 isvisible above the mounting bracket 1250. The longitudinal member 1240may be rotated about a center of rotation 1254 defined by thelongitudinal member bearings 1252. The end effector 1210 may further berotated relative to the radial member 1220 about a second rotationalaxis 1207 b defined by the center of the mounting plate 1212, controlledby motion of the end effector actuator gear 1214. Rotational motion 1206b illustrates the rotation of the end effector relative to the radialmember 1220 about the second rotational axis. In some embodiments, themotion 1206 a, 1206 b about the first rotational axis 1207 a and thesecond rotational axis 1207 b may be linked such that clockwise motion1206 a, 1206 b about either rotational axis is accompanied bycounterclockwise motion 1206 a, 1206 b about the other rotational axis1207 a, 1207 b. In such embodiments, the linking of the rotational axes1207 a, 1207 b allows the end effector 1210 to be retained in the sameorientation relative to the ground (e.g., vertical or substantiallyvertical) as the radial member 1220 rotates about the first rotationalaxis 1207 a. Rotational motion 1206 a, 1206 b about either rotationalaxis 1207 a, 1207 b may be driven by a motor 1256 mounted to thecarriage 1230.

With reference to FIG. 12D, in some embodiments the end effector 1210may further be rotatable about a third rotational axis 1209 parallel tothe radial member 1220, as shown by rotational motion 1208. For example,the distal end 1224 of the radial member 1220 may be rotatable relativeto the remainder (e.g., a proximal portion) of the radial member 1220.In embodiments in which motion 1206 a, 1206 b about the first and secondrotational axes 1207 a, 1207 b are not linked, motion 1208 about thethird rotational axis 1209 may be used in combination with motion 1206 babout the second rotational axis 1207 b to position the end effector1210 at any orientation including sideways and/or forward/backwardorientations. Such additional possible orientations may allow for thepicking of items that are not easily accessible and/or visible fromabove, but which are accessible and/or visible from the side, in front,and/or behind.

With continued reference to FIG. 12D, the end effector 1210 may bemounted to the mounting plate 1212 by an end effector mounting bracket1216. Rotation of the end effector 1210 about the second rotational axis1207 b may be controlled by an end effector rotation motor 1218 coupledto the distal end 1224 of the radial member 1220.

FIGS. 12E-12G illustrate example methods of moving the end effector 1210using the robot 1200 with combined motion along the radial axis 1205 andabout the first and second rotational axes 1207 a, 1207 b. It will beunderstood that the linear axes 1203 and 1205, and the first rotationalaxis 1207 a, together form a cylindrical coordinate system. For a givenposition of the carriage 1230 along the longitudinal axis 1203, motionof the end effector 1210 occurs within a two-dimensional polarcoordinate system in which the position of the end effector 1210 can bedefined by a set of coordinates (r, θ), where r is defined by thedisplacement of the radial member 1220 relative to the carriage 1230,and θ is defined by the rotational position of the longitudinal member1240 about the first rotational axis 1207 a.

FIG. 12E depicts an initial position of the system in which the endeffector 1210 is positioned generally below and slightly offset to theright relative to the mounting brackets 1250 and the longitudinal member1240. To transition from the position of FIG. 12E to the position ofFIG. 12F, the end effector 1210 moves directly upward and retains avertical orientation. Such motion would be characterized in a Cartesiancoordinate system as a change in the y-coordinate with a constantx-coordinate. In the polar coordinate system of the robot 1200, the samevertical motion is achieved by moving the radial member 1220 relative tothe carriage 1230 such that r decreases, and rotating the carriage 1230and radial member 1220 counterclockwise about the center of rotation1254, as shown in FIG. 12F. As the carriage 1230 and radial member 1220are rotated to increase θ, the end effector 1210 may also be rotatedclockwise at the end effector actuator gear 1214 to maintain the endeffector 1210 in a vertical orientation.

From the position shown in FIG. 12F, the system may then transition tothe position of FIG. 12G. To transition to the position of FIG. 12G, theend effector 1210 moves directly to the right and retains a verticalorientation. Such motion would be characterized in a Cartesiancoordinate system as a change in the x-coordinate with a constanty-coordinate. In the polar coordinate system of the robot 1200, the samevertical motion is achieved by moving the radial member 1220 relative tothe carriage 1230 such that r increases, and rotating the carriage 1230and radial member 1220 further counterclockwise about the center ofrotation 1254, as shown in FIG. 12G. As the carriage 1230 and radialmember 1220 are rotated to further increase θ, the end effector 1210 mayalso be rotated further clockwise at the end effector actuator gear 1214to maintain the end effector 1210 in a vertical orientation.

In one example embodiment, an autonomous picking device comprises an endeffector configured to pick items and a robot coupled to the endeffector. The robot comprises a longitudinal member 1240 defining afirst rotational axis 1207 a extending along a length of thelongitudinal member, the longitudinal member configured to rotate 1206 awithin the picking device about the first rotational axis, thelongitudinal member further defining a first translational axisextending between a first end and a second end of the longitudinalmember; a carriage 1230 defining a second translational axis 1205extending between a first end and a second end of the carriage, thesecond translational axis perpendicular to the first translational axisof the longitudinal member, the carriage configured to translate alongthe first translational axis of the longitudinal member; a radial member1220 defining a second rotational axis 1207 b perpendicular to a lengthof the radial member at a distal end of the radial member, the radialmember configured to translate along the second translational axis ofthe carriage; and a rotatable end effector mount coupling the endeffector to the distal end of the radial member, the end effectorconfigured to rotate 1206 b within the picking device about the secondrotational axis of the radial member.

Example Harvester According to the Present Disclosure

FIGS. 13A-13D depict a further example harvester 1300 in accordance withthe systems and methods described herein. FIG. 13A is an upper, right,rear perspective view of the harvester 1300. FIG. 13B is a lower, left,front perspective view of the harvester 1300. FIG. 13C is a rear view ofthe harvester 1300. FIG. 13D is a side view of the harvester 1300.Referring jointly to FIGS. 13A-13D, the harvester 1300 is configured forharvesting berries or other items from a plurality of planting beds asthe harvester 1300 travels along a forward direction 1301. The harvester1300, or any component thereof, may be implemented in conjunction withany of the embodiments or aspects illustrated herein and described withreference to FIGS. 1-12G. Although the harvester 1300 will be describedwith reference to a particular set of features contained herein, theharvester 1300 is not limited to the particular components depicted inFIGS. 13A-13D, and may equally be implemented with fewer components,additional components, or different components relative to those ofFIGS. 13A-13D.

The harvester 1300 includes a chassis including a longitudinal member1302 and a lateral member 1304, a body 1310, drive wheel units 1320,robots 1200 _(i), 1200 _(o), and container handling units 1330 _(i),1330 _(o). The body 1310, the drive wheel units 1320, and the containerhandling units 1330 _(i), 1330 _(o), as well as an air compressor 1303,a forward wheel 1305, a harvester control panel 1306, a generator fueltank 1307, a vehicle power supply 1308, and a generator 1309, may becoupled to the longitudinal member 1302 and/or the lateral member 1304of the chassis.

The body 1310 comprises a frame surrounding the work cell area includingthe robots 1200 _(i), 1200 _(o). The body 1310 may be a generally openframe, or may be enclosed by a skin 1312 to protect components such asthe robots 1200 _(i), 1200 _(o) in the work cell area (e.g., from sun,precipitation, etc.). In some embodiments, a rear extension 1314provides protection from sun or precipitation for items that have beenpicked and are being stored in containers at the rear of the harvester1300.

Drive wheel units 1320 are mounted to the lateral member 1304 of thechassis and are configured to support and propel the harvester 1300.Each drive wheel unit 1320 includes a wheel 1322, a mounting arm 1324, adrive motor 1326, and a drive motor controller 1328. The drive wheelunits 1320 are in wired or wireless communication with the harvestercontrol panel 1306.

The chassis further supports a plurality of robots 1200 _(i), 1200 _(o),including two inboard robots 1200 _(i) mounted medially near thelongitudinal member 1302, and two outboard robots 1200 _(o) mountedlaterally outboard from the inboard robots 1200 _(i). Each robot 1200_(i), 1200 _(o) supports and moves an end effector 1210 _(i), 1210 _(o).As described elsewhere herein, all four of the robots 1200 _(i), 1200_(o) may operate independently and simultaneously as the harvester 1300travels along one or more rows of plants, and may be controlled so as toavoid collisions between adjacent robots 1200 _(i), 1200 _(o).

Container handling units 1330 _(i), 1330 _(o) are coupled to the lateralmember 1304 of the chassis such that each container handling unit 1330_(i), 1330 _(o) is aft of and substantially aligned with one of therobots 1200 _(i), 1200 _(o). Each container handling unit 1330 _(i),1330 _(o) is configured to hold a stack of empty containers 1332 and/ora stack of filled containers 1334. Containers 1332, 1334 may be plastic,metal, cardboard, or other single-use or reusable containers. Thecontainer handling units 1330 _(i), 1330 _(o) are configured to moveindividual containers, such as an active container 1338, along rails1336 between the stacks of containers 1332, 1334 and a forward position,as shown in FIG. 13D.

In one example process of operating the harvester 1300, a stack of emptycontainers 1332, such as reusable plastic containers or trays, is placedinto each of the four container handling units 1330 _(i), 1330 _(o). Theharvester 1300 is then positioned to span two beds with the drive wheelunits 1320 positioned within the furrows on the outer sides of the twobeds and the forward wheel 1305 positioned within the furrow between thetwo beds. The harvester 1300 may be driven forward until the forwardwheel 1305 is aligned with a first plant along one or both beds. Anautonomous driving mode may be engaged. As the harvester 1300 beginsmoving forward under the power of the drive wheel units 1320, eachcontainer handling unit 1330 _(i), 1330 _(o) drops a first emptycontainer from the bottom of the stack of empty containers 1332 onto therails 1336 to become the active container 1338. One or more motors (notshown) move the active container 1338 forward, toward a front end of therails 1336.

In some embodiments, each container handling unit 1330 _(i), 1330 _(o)includes an optical sensor (e.g., a photo eye, light gate, proximitysensor, etc.), coupled to the rails 1336 and spaced aft of the front endof the rails 1336 by a distance approximately equal to the width of thecontainers 1332, 1334. As the front end of the active container 1338reaches the optical sensor, the optical sensor asserts due to thepresence of the wall of the active container 1338 at the optical sensor.As the active container 1338 continues moving forward, the rear end ofthe active container 1338 passes the optical sensor, causing the opticalsensor to deassert. Deassertion of the optical sensor indicates to thecontainer handling unit 1330 _(i), 1330 _(o) that the active container1338 has reached the full forward position (e.g., the positionillustrated in FIG. 13D), and causes the one or more motors to stopmoving the active container 1338 forward. In some embodiments, theoptical sensor can further be configured to detect a jam or malfunctionof the tray handling unit 1330 _(i), 1330 _(o), such as if the opticalsensor asserts and fails to deassert after a predetermined thresholdtime period.

When the active tray 1338 has reached the full forward position, thecorresponding robot 1200 _(i), 1200 _(o) begins or continues pickingitems from the bed below, and deposits the picked items into the activecontainer 1338. Each robot 1200 _(i), 1200 _(o) continues picking itemsuntil a processing component of the harvester 1300 determines that theactive container 1338 is full. The active container 1338 may bedetermined to be full by various methods, for example, based on a weightchange of the active container 1338, based on an optically detectedlevel of items resting in the active container 1338, and/or based on anitem count. For example, the harvester 1300 may be configured such thateach active container 1338 is determined to be full when thecorresponding robot 1200 _(i), 1200 _(o) has picked a predeterminednumber of items since the current active container 1338 was positioned.The predetermined number may be selected based on an empiricallydetermined average size of the items or average number of items neededto fill a container.

When the active container 1338 is determined to be full, thecorresponding robot 1200 _(i), 1200 _(o) temporarily stops depositingitems into the active container 1338. The one or more motors move theactive container 1338 rearward to the location of the stack of filledcontainers 1334. After the active container 1338 passes the stack ofempty containers 1332, the bottom container of the stack of emptycontainers 1332 is dropped onto the rails 1336, where the bottomcontainer becomes the new active container 1338 and is moved forward toreceive additional items picked by the corresponding robot 1200 _(i),1200 _(o). The process repeats, with each filled active container 1338being moved backward when full to form the new bottom tray of the stackof filled containers 1334 and a new active container 1338 being loweredfrom the stack of empty containers 1332. When all empty trays 1332 havebeen filled, all of the containers in the harvester 1300 are full andare located within one of the stacks of filled containers 1334, whichare located at the rear of the harvester 1300 such that they may beeasily removed by an attendant.

In some embodiments, the harvester 1300 is at least partially modularsuch that it can be reconfigured as desired to pick items from fieldshaving different bed widths. For example, some strawberry growingoperations are implemented with beds having a width of 64 inches, whileother strawberry growing operations use beds having a width of 48inches. In the configuration illustrated in FIGS. 13A-13D, the wheels1322 of the drive wheel units 1320 may each be offset laterally relativeto the forward wheel 1305 by approximately 64 inches, corresponding tothe distance between adjacent furrows in a field having 64-inch beds. Inthis configuration, the harvester 1300 can pick items from two adjacent64-inch beds simultaneously, with two of the robots 1200 _(i), 1200 _(o)disposed above and picking from each bed. However, this configurationwould not be suitable for a field having 48-inch beds, because thewheels 1322 of the drive wheel units 1320 would need to be offsetrelative to the forward wheel 1305 by either 48 inches or 96 inches inorder to rest in a furrow.

In order to pick items from 48-inch beds, the harvester 1300 may bereconfigured by removing the drive wheel units 1320, the outboard robots1200 _(o), and the outboard container handling units 1330 _(o). Theinboard robots 1200 _(i) and the inboard container handling units 1330_(i) may be left in place. After removing the outboard components, thedrive wheel units 1320 may be reattached adjacent to the inboardcontainer handling units 1330 _(i) such that the wheels 1322 arelaterally offset relative to the forward wheel 1305 by approximately 48inches. The outboard container handling units 1330 _(o) may then bereattached outboard of the drive wheel units 1320, and the outboardrobots 1200 _(o) may be reattached such that they are substantiallyaligned with the outboard container handling units 1330 _(o). In thisconfiguration, the harvester 1300 can pick items from four adjacent48-inch beds simultaneously, with one of the robots 1200 _(i), 1200 _(o)disposed above and picking from each bed. Such reconfiguration of theharvester 1300 to service fields having variable bed widths, isfacilitated by the use of self-contained drive wheel units 1320 whichmay easily be detached and attached as standalone units.

Advantages of Implementations of the Present Disclosure

Without limiting the scope of the foregoing description, additionaladvantageous features of certain embodiments of the present disclosurewill now be described.

Some embodiments may be advantageously adapted for picking of delicateitems such as berries or other agricultural products. If these objectsare bruised, scratched, discolored, or otherwise harmed by the pickingprocess, they become nearly worthless. In some aspects, features such asresilient or cushioned gripper fingers, grasp force sensing, anduntensioning and rotating during a harvesting process as describedherein, may reduce the amount of force exerted against items and therebyreduce the probability of damaging the items during harvesting. Suchharvesting without damaging the items is especially advantageous in theexample of berries or other objects that must be disengaged from atether (for example, a stem), rather than simply being picked up.

Real-time control, termination, and/or modification of harvestingprocesses may also be advantageous. As described above, a pick may beunsuccessful if a berry is not removed from the plant or subsequentlydropped. However, certain real-time control methods described herein maypermit dropped berries to be re-picked. A dropped berry may remainwithin the work volume, and can be picked later by a second pass of therobot or a different robot, or by hand. In some cases, a dropped berrymay be immediately re-attempted by moving the end effector upward by ashort distance (e.g., a few inches), detecting and localizing the berryagain, and trying again from the short distance, rather than returningto a high-level scan state.

In some embodiments, the harvester may enhance efficiency by not tryingto pick every possible berry within the work volume. Embodiments of thepresently-disclosed harvesters can be configured to attempt to pick allpossible berries, but in many cases it may be intended to achieve aparticular efficiency or pick rate. For example, a harvester may bedesigned to pick a certain percentage of candidate strawberries lessthan 100% to ensure that the strawberries are picked in an optimalharvest window. Berries that are not picked may be recovered later. Inanother example, the harvester may pick a percentage of fruit that is ina particular range of ripeness, which may be user-adjustable such thatthe user may decide which fruit is most desirable to harvest.

In some embodiments, the parameters of a candidate berry can be adjustedto take into account many different considerations. In some non-limitingexamples, some plants are more dense and hide strawberries better, olderplants tend to have longer stems that can be broken more easily, thefield may need to be picked within a specified time frame forenvironmental or cost reasons, a user may desire a particular yield orto only pick berries that meet higher quality standards, a user maydesire to clear all strawberries in the field because they are diseasedor otherwise undesirable, or any combination of these conditions andparameters may exist. In one particular example, a harvester may be usedto quickly pick the best berries in a field before the arrival of anincoming freeze, storm, or other environmental event.

In some embodiments, harvesters may be configured to prioritize thehealth and survival of the plants. As described above with reference toFIGS. 1-3, some embodiments of harvesters described herein areconfigured to be supported within furrows between rows of plants, ratherthan traveling in contact with the rows. This configuration, in whichthe harvesting components are suspended above the plants, preventstrampling or other damage that may prevent the plants from producingadditional fruit in the future. In another example, certain steps of theharvesting processes described above may further promote plant health.For example, rotating the berry before the final separation from thestem may protect not just the berry from damage, but also the plant, dueto the reduction in force required to separate the berry from the stem.

Further Use Cases

It will be appreciated that the systems and methods described herein arenot limited to the context of picking horizontally-grown strawberries orother fruit that are positioned in a horizontally-grown plant in thesame or similar orientation as strawberry plants. Rather, theharvesters, work cells, robots, end effectors, and all componentsthereof, as described herein, may be use for a wide variety ofimplementations.

In one example, grapes or other vine crops, vertically-grownstrawberries, and the like may be similarly picked and/or harvestedusing the systems and methods described herein. Because vine cropsgenerally grow in a substantially vertical orientation, such harvestersmay be configured such that the robots to which the end effectors aremounted are located in a horizontal, rather than a vertical,orientation. For example, the robots of FIGS. 12A-12G may be mounted ina position rotated by approximately 90 degrees about the longitudinalmember relative to the orientation depicted herein, such that they canmanipulate one or more end effectors to pick items from a substantiallyvertical picking environment. The computer vision systems describedherein may be configured to identify vine crops based on color, shaped,bunch shape, or other criteria to reliably identify grapes or other vinecrops to be picked. In some embodiments, end effectors configured forpicking of vine crops may further include one or more blades or othercutting or snipping elements in order to allow for snipping of bunchesrather than harvesting of individual grapes or other vine fruits.

In some embodiments, blades or other cutting or snipping elements mayfurther be employed for pruning of various crops. For example, treesproducing almonds, walnuts, peaches, cherries, or other tree crops mayrequire pruning which may easily be accomplished using the systemsdisclosed herein.

Other ground crops may further be harvested using the systems andmethods of the present disclosure. For example, cucumbers, watermelons,other melons, broccoli, cauliflower, asparagus, and the like, may all beharvested by the harvesters described herein. In some embodiments,modifications to the disclosed end effectors may include changes in thesize and placement of fingers, gripping forces, etc., as required toreliably harvest each individual type of ground crop.

Implementing Systems and Terminology

Implementations disclosed herein provide systems, methods, and devicesfor autonomous selection and retrieval of items using robots. Oneskilled in the art will recognize that these embodiments may beimplemented in hardware or a combination of hardware and software and/orfirmware.

Embodiments of robots according to the present disclosure may includeone or more sensors (for example, image sensors), one or more signalprocessors (for example, image signal processors), and a memoryincluding instructions or modules for carrying out the processesdiscussed above. The robot may also have data, a processor loadinginstructions and/or data from memory, one or more communicationinterfaces, one or more input devices, one or more output devices suchas a display device and a power source/interface. The device mayadditionally include a transmitter and a receiver. The transmitter andreceiver may be jointly referred to as a transceiver. The transceivermay be coupled to one or more antennas for transmitting and/or receivingwireless signals.

The functions described herein may be stored as one or more instructionson a processor-readable or computer-readable medium. The term“computer-readable medium” refers to any available medium that can beaccessed by a computer or processor. By way of example, and notlimitation, such a medium may comprise RAM, ROM, EEPROM, flash memory,CD-ROM or other optical disk storage, magnetic disk storage or othermagnetic storage devices, or any other medium that can be used to storedesired program code in the form of instructions or data structures andthat can be accessed by a computer. It should be noted that acomputer-readable medium may be tangible and non-transitory. The term“computer-program product” refers to a computing device or processor incombination with code or instructions (e.g., a “program”) that may beexecuted, processed or computed by the computing device or processor. Asused herein, the term “code” may refer to software, instructions, codeor data that is/are executable by a computing device or processor.

The various illustrative logical blocks and modules described inconnection with the embodiments disclosed herein can be implemented orperformed by a machine, such as a general purpose processor, a digitalsignal processor (DSP), an application specific integrated circuit(ASIC), a field programmable gate array (FPGA) or other programmablelogic device, discrete gate or transistor logic, discrete hardwarecomponents, or any combination thereof designed to perform the functionsdescribed herein. A general purpose processor can be a microprocessor,but in the alternative, the processor can be a controller,microcontroller, or state machine, combinations of the same, or thelike. A processor can also be implemented as a combination of computingdevices, e.g., a combination of a DSP and a microprocessor, a pluralityof microprocessors, one or more microprocessors in conjunction with aDSP core, or any other such configuration. Although described hereinprimarily with respect to digital technology, a processor may alsoinclude primarily analog components. For example, any of the signalprocessing algorithms described herein may be implemented in analogcircuitry. A computing environment can include any type of computersystem, including, but not limited to, a computer system based on amicroprocessor, a mainframe computer, a digital signal processor, aportable computing device, a personal organizer, a device controller,and a computational engine within an appliance, to name a few.

The foregoing description details certain embodiments of the systems,devices, and methods disclosed herein. It will be appreciated, however,that no matter how detailed the foregoing appears in text, the systems,devices, and methods can be practiced in many ways. It should be notedthat the use of particular terminology when describing certain featuresor aspects of the present disclosure should not be taken to imply thatthe terminology is being re-defined herein to be restricted to includingany specific characteristics of the features or aspects of thetechnology with which that terminology is associated.

It will be appreciated by those skilled in the art that variousmodifications and changes may be made without departing from the scopeof the described technology. Such modifications and changes are intendedto fall within the scope of the embodiments. It will also be appreciatedby those of skill in the art that parts included in one embodiment areinterchangeable with other embodiments; one or more parts from adepicted embodiment can be included with other depicted embodiments inany combination. For example, any of the various components describedherein and/or depicted in the Figures may be combined, interchanged orexcluded from other embodiments.

The methods disclosed herein include one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isrequired for proper operation of the method that is being described, theorder and/or use of specific steps and/or actions may be modifiedwithout departing from the scope of the claims.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for sakeof clarity.

It will be understood by those within the art that, in general, termsused herein are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). Further, the term “comprising” as used herein is synonymous with“including,” “containing,” or “characterized by,” and is inclusive oropen-ended and does not exclude additional, unrecited elements or methodsteps. Accordingly, the term “comprising”, used in the claims, shouldnot be interpreted as being restricted to the means listed thereafter;it does not exclude other elements or steps. It is thus to beinterpreted as specifying the presence of the stated features, integers,steps or components as referred to, but does not preclude the presenceor addition of one or more other features, integers, steps orcomponents, or groups thereof. Thus, the scope of the expression “adevice comprising means A and B” should not be limited to devicesconsisting only of components A and B. It will be further understood bythose within the art that virtually any disjunctive word and/or phrasepresenting two or more alternative terms, whether in the description,claims, or drawings, should be understood to contemplate thepossibilities of including one of the terms, either of the terms, orboth terms. For example, the phrase “A or B” will be understood toinclude the possibilities of “A” or “B” or “A and B.”

It should be noted that the terms “couple,” “coupling,” “coupled” orother variations of the word couple as used herein may indicate eitheran indirect connection or a direct connection. For example, if a firstcomponent is “coupled” to a second component, the first component may beeither indirectly connected to the second component or directlyconnected to the second component. As used herein, the term “plurality”denotes two or more. For example, a plurality of components indicatestwo or more components.

The term “determining” encompasses a wide variety of actions and,therefore, “determining” can include calculating, computing, processing,deriving, investigating, looking up (e.g., looking up in a table, adatabase or another data structure), ascertaining and the like. Also,“determining” can include receiving (e.g., receiving information),accessing (e.g., accessing data in a memory) and the like. Also,“determining” can include resolving, selecting, choosing, establishingand the like. The phrase “based on” does not mean “based only on,”unless expressly specified otherwise. In other words, the phrase “basedon” describes both “based only on” and “based at least on.”

It is noted that some examples above may be described as a process,which is depicted as a flowchart, a flow diagram, a structure diagram,or a block diagram. Although a flowchart may describe the operations asa sequential process, many of the operations can be performed inparallel, or concurrently, and the process can be repeated. In addition,the order of the operations may be rearranged. A process is terminatedwhen its operations are completed. A process may correspond to a method,a function, a procedure, a subroutine, a subprogram, etc. When a processcorresponds to a software function, its termination corresponds to areturn of the function to the calling function or the main function.

The above description discloses several methods and materials of thepresent disclosure. Embodiments of the present disclosure aresusceptible to modifications in the methods and materials, as well asalterations in the fabrication methods and equipment. Such modificationswill become apparent to those skilled in the art from a consideration ofthis disclosure or practice of the present disclosure. Consequently, itis not intended that the present disclosure be limited to the specificembodiments disclosed herein, but that it cover all modifications andalternatives coming within the true scope and spirit of the presentdisclosure as embodied in the attached claims.

1. An end effector for a picking robot, the end effector comprising: ashaft extending along a longitudinal axis between a proximal end and adistal end; a carriage configured to rotate about and translate along anintermediate portion of the shaft between a proximal position and adistal position; a suction device coupled to the distal end of theshaft; and a plurality of fingers spaced radially about the carriage,each finger coupled to the carriage by a hinge and extending distallyfrom the carriage, the plurality of fingers configured to envelop theintermediate portion of the shaft when the carriage is in the proximalposition, the plurality of fingers further configured to envelop atarget object space comprising the distal end of the shaft and thesuction device when the carriage is in the distal position.
 2. The endeffector of claim 1, wherein the suction device is configured to apply avacuum to the target object space when the carriage is in the proximalposition and in the distal position.
 3. The end effector of claim 1,wherein the suction device is a distal-most portion of the end effectorwhen the carriage is in the proximal position, and wherein the pluralityof fingers are the distal-most portion of the end effector when thecarriage is in the distal position.
 4. The end effector of claim 1,wherein the plurality of figures do not envelop the distal end of theshaft when the carriage is in the proximal position.
 5. The end effectorof claim 1, wherein the plurality of fingers are configured to envelopan object within the target object space when the carriage is in thedistal position.
 6. The end effector of claim 1, wherein each of theplurality of fingers is configured to rotate about the hinge toward thelongitudinal axis of the shaft.
 7. The end effector of claim 6, whereina volume of the target object space decreases when each of the pluralityof fingers rotates toward the longitudinal axis of the shaft.
 8. The endeffector of claim 1, wherein the carriage is rotationally fixed aboutthe shaft, and wherein the shaft is rotatably mounted within the endeffector such that the carriage is rotatable by rotating the shaft aboutthe longitudinal axis.
 9. The end effector of claim 1, wherein the shaftcomprises a collar extending radially outward about the distal end ofthe shaft, wherein each finger comprises a finger extension extendingradially inward toward the longitudinal axis at a proximal end of thefinger, and wherein, as the carriage moves from the proximal positiontowards the distal position, the finger extension is configured tocontact the collar before the carriage reaches the distal position. 10.The end effector of claim 9, wherein movement of the carriage to thedistal position following contact of each finger extension with thecollar causes each finger to rotate about the hinge toward thelongitudinal axis.
 11. The end effector of claim 9, wherein movement ofthe carriage to the distal position following contact of each fingerextension with the collar causes the volume of the target object spaceto decrease.
 12. The end effector of claim 1, wherein each fingercomprises a force sensor configured to detect a force exerted betweenthe finger and an object within the target object space.
 13. The endeffector of claim 12, wherein the force exerted between the finger andthe object comprises at least one of a normal force and a shear force.14. The end effector of claim 1, wherein the suction device comprises atleast one sensor configured to detect the presence of an object withinthe target object space.
 15. The end effector of claim 14, wherein theat least one sensor is selected from the group consisting of a linearposition sensor and a vacuum sensor.
 16. The end effector of claim 14,wherein the at least one sensor comprises a linear position sensor or a3D force sensor, wherein the object within the target object space istethered to a stationary object, and wherein the at least one sensor isconfigured to detect a direction at which the object is tethered to thestationary object.
 17. The end effector of claim 1, further comprising acamera coupled to the end effector and processing circuitry configuredto detect target objects based at least in part on images obtained fromthe camera.
 18. The end effector of claim 17, wherein the processingcircuitry is further configured to determine target object locationswith respect to a coordinate system based at least in part on the imagesobtained from the camera.
 19. The end effector of claim 18, wherein theprocessing circuitry is further configured to share at least one ofimage data and target object location data with processing circuitryassociated with a second end effector.
 20. The end effector of claim 1,wherein the suction device is a suction cup comprising a collapsiblebellows.
 21. The end effector of claim 1, wherein the end effector iscoupled to a picking robot of a movable harvester.
 22. The end effectorof claim 21, wherein the harvester further comprises a LIDAR systemconfigured to guide the harvester along a path parallel to a furrow ofan agricultural field based on detection of at least a portion of aprofile of the furrow. 23-44. (canceled)